==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== 131m Warning FailedToRetrieveImagePullSecret pod/maas-controller-656b6867cb-5b9h9 Unable to retrieve some image pull secrets (maas-controller-dockercfg-7z2ct); attempting to pull the image may not succeed. 127m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 127m Warning FailedCreate replicaset/spark-operator-controller-57c5867998 Error creating: pods "spark-operator-controller-57c5867998-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 127m Warning BackOff pod/spark-operator-controller-57c5867998-szrpd Back-off restarting failed container controller in pod spark-operator-controller-57c5867998-szrpd_redhat-ods-applications(17ccf8a3-035b-4071-866a-0326e08f40ad) 127m Warning FailedToRetrieveImagePullSecret pod/spark-operator-controller-57c5867998-szrpd Unable to retrieve some image pull secrets (spark-operator-controller-dockercfg-czkf7); attempting to pull the image may not succeed. 127m Warning FailedMount pod/spark-operator-webhook-56c5bb987f-kswlc MountVolume.SetUp failed for volume "kube-api-access-2xwh5" : failed to fetch token: serviceaccounts "spark-operator-webhook" not found 126m Warning FailedMount pod/model-registry-operator-controller-manager-5f68f4fd76-d2jqr MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 126m Warning FailedMount pod/odh-model-controller-6b647f87c-gbfdf MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 126m Warning FailedCreate replicaset/kserve-controller-manager-6c8cb86d7f Error creating: pods "kserve-controller-manager-6c8cb86d7f-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 126m Warning FailedCreate replicaset/spark-operator-controller-57c5867998 Error creating: pods "spark-operator-controller-57c5867998-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 126m Warning FailedCreate replicaset/llmisvc-controller-manager-586c85c465 Error creating: pods "llmisvc-controller-manager-586c85c465-" is forbidden: error looking up service account redhat-ods-applications/llmisvc-controller-manager: serviceaccount "llmisvc-controller-manager" not found 126m Warning FailedMount pod/ogx-k8s-operator-controller-manager-67866ffc79-glpwq MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 126m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 126m Warning FailedMount pod/odh-notebook-controller-manager-7f6b58f576-gskzh MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 126m Warning FailedCreate replicaset/notebook-controller-deployment-5d7d79cf5b Error creating: pods "notebook-controller-deployment-5d7d79cf5b-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 126m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 125m Warning CalculateExpectedPodCountFailed poddisruptionbudget/ogx-k8s-operator-controller-manager-pdb Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-67866ffc79-glpwq" 125m Warning CalculateExpectedPodCountFailed poddisruptionbudget/ogx-k8s-operator-controller-manager-pdb Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-67866ffc79-rdjtg" 125m Warning Unhealthy pod/ogx-k8s-operator-controller-manager-67866ffc79-glpwq Readiness probe failed: Get "http://10.130.3.41:8081/readyz": dial tcp 10.130.3.41:8081: connect: connection refused 125m Warning FailedMount pod/spark-pi-d95lqfp48los74uh76b0-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-2a36089f36647321-conf-map" not found 125m Warning FailedMount pod/spark-pi-d95lqfp48los74uh76b0-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d95lqfp48los74uh76b0-77655d9f36645fa1-driver-podspec-conf-map" not found 125m Warning FailedToUpdateEndpoint endpoints/data-science-pipelines-operator-service Failed to update endpoint redhat-ods-applications/data-science-pipelines-operator-service: Operation cannot be fulfilled on endpoints "data-science-pipelines-operator-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/data-science-pipelines-operator-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 96541ebd-3e08-4f82-932c-ccc33d6fcb27, UID in object meta: 125m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-7fc7f76n57jb MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 125m Warning FailedMount pod/kuberay-operator-6b449f854b-v9wcc MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 125m Warning FailedCreate replicaset/rhods-dashboard-6b5cc9785c Error creating: pods "rhods-dashboard-6b5cc9785c-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 124m Warning FailedCreate replicaset/rhods-dashboard-6b5cc9785c Error creating: pods "rhods-dashboard-6b5cc9785c-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 124m Warning FailedMount pod/rhods-dashboard-6b5cc9785c-v7hkz MountVolume.SetUp failed for volume "modules-sa-token" : references non-existent secret key: token 124m Warning Failed pod/rhods-dashboard-6b5cc9785c-v7hkz Error: ErrImagePull 124m Warning Failed pod/rhods-dashboard-6b5cc9785c-v7hkz Failed to pull image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:88aa0c68b57cc272713ec1f909559adbeec11e509457c3d4843a49797567261b": initializing source docker://registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:88aa0c68b57cc272713ec1f909559adbeec11e509457c3d4843a49797567261b: (Mirrors also failed: [brew.registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:88aa0c68b57cc272713ec1f909559adbeec11e509457c3d4843a49797567261b: reading manifest sha256:88aa0c68b57cc272713ec1f909559adbeec11e509457c3d4843a49797567261b in brew.registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9: name unknown: Digest not found]... 124m Warning Failed pod/rhods-dashboard-6b5cc9785c-v7hkz Error: ImagePullBackOff 124m Warning FailedMount pod/scheduled-spark-pi-d95lqsh48los74uh76bg-1783324048001419356-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d95lqsh48los74uh76bg-1783324048001419356-1fb70c9f3665975b-driver-podspec-conf-map" not found 124m Warning FailedMount pod/scheduled-spark-pi-d95lqsh48los74uh76bg-1783324048001419356-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-6712f59f3665aa0d-conf-map" not found 124m Warning FailedToUpdateEndpointSlices service/data-science-pipelines-operator-service Error updating Endpoint Slices for Service redhat-ods-applications/data-science-pipelines-operator-service: failed to delete data-science-pipelines-operator-service-84mtw EndpointSlice for Service redhat-ods-applications/data-science-pipelines-operator-service: endpointslices.discovery.k8s.io "data-science-pipelines-operator-service-84mtw" not found 115m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 115m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 115m Warning InvalidConfiguration servicemonitor/workload-variant-autoscaler-controller-manager-metrics-monitor ServiceMonitor workload-variant-autoscaler-controller-manager-metrics-monitor was rejected due to invalid configuration: failed to get authorization token of type "": unable to get secret "workload-variant-autoscaler-metrics-reader-token": secrets "workload-variant-autoscaler-metrics-reader-token" not found 112m Warning FailedCreate replicaset/kserve-controller-manager-6c8cb86d7f Error creating: pods "kserve-controller-manager-6c8cb86d7f-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 109m Warning FailedMount pod/odh-model-controller-54bf759f64-kjt65 MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 109m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 108m Warning FailedMount pod/mlflow-operator-controller-manager-8646f94cf4-dsp7p MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 108m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 108m Warning FailedMount pod/maas-api-84c4b8dcb6-gb94r MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-serving-cert" not found 107m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-6f7f88c874 Error creating: pods "trustyai-service-operator-controller-manager-6f7f88c874-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 107m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning FailedMount pod/odh-model-controller-54bf759f64-22kmw MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 105m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 104m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 104m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning FailedToRetrieveImagePullSecret pod/llmisvc-controller-manager-586c85c465-hcdk8 Unable to retrieve some image pull secrets (llmisvc-controller-manager-dockercfg-x2zz7); attempting to pull the image may not succeed. 103m Warning FailedToRetrieveImagePullSecret pod/kserve-controller-manager-6c8cb86d7f-8ngts Unable to retrieve some image pull secrets (kserve-controller-manager-dockercfg-7zr65); attempting to pull the image may not succeed. 103m Warning FailedMount pod/odh-model-controller-54bf759f64-76pvs MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 103m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 103m Warning FailedMount pod/model-serving-api-78bbbb9bf8-4xzmv MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 100m Warning FailedMount pod/model-registry-operator-controller-manager-5f68f4fd76-skcfc MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 100m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-7fc7f768f5wq MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 100m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-6f7f88c874 Error creating: pods "trustyai-service-operator-controller-manager-6f7f88c874-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 100m Warning FailedMount pod/mlflow-operator-controller-manager-8646f94cf4-pmz5p MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 100m Warning FailedMount pod/ogx-k8s-operator-controller-manager-67866ffc79-sjnxz MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 100m Warning FailedMount pod/odh-notebook-controller-manager-7f6b58f576-pwk7q MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 100m Warning FailedCreate replicaset/spark-operator-controller-57c5867998 Error creating: pods "spark-operator-controller-57c5867998-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 100m Warning FailedCreate replicaset/notebook-controller-deployment-5d7d79cf5b Error creating: pods "notebook-controller-deployment-5d7d79cf5b-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 100m Warning ErrorUpdatingResource pod/notebook-controller-deployment-5d7d79cf5b-gb4cr addLogicalPort failed for redhat-ods-applications/notebook-controller-deployment-5d7d79cf5b-gb4cr: failed to update pod redhat-ods-applications/notebook-controller-deployment-5d7d79cf5b-gb4cr: Operation cannot be fulfilled on pods "notebook-controller-deployment-5d7d79cf5b-gb4cr": the object has been modified; please apply your changes to the latest version and try again 100m Warning BackOff pod/trustyai-service-operator-controller-manager-6f7f88c874-5gk7q Back-off restarting failed container manager in pod trustyai-service-operator-controller-manager-6f7f88c874-5gk7q_redhat-ods-applications(afa97808-57db-46d5-8a99-ecb3ec30b632) 100m Warning FailedCreate replicaset/rhods-dashboard-6b5cc9785c Error creating: pods "rhods-dashboard-6b5cc9785c-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 100m Warning FailedMount pod/rhods-dashboard-6b5cc9785c-mpn5c MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 100m Warning FailedMount pod/rhods-dashboard-6b5cc9785c-2qxdw MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 100m Warning FailedMount pod/kuberay-operator-6b449f854b-mxlmx MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 99m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 96m Warning Unhealthy pod/mlflow-668464596d-sxm9n Readiness probe failed: Get "https://10.131.2.162:8443/mlflow/health": dial tcp 10.131.2.162:8443: connect: connection refused 94m Warning Unhealthy pod/mlflow-58f77fb56c-rpv9n Readiness probe failed: Get "https://10.131.2.165:8443/mlflow/health": dial tcp 10.131.2.165:8443: connect: connection refused 6m22s Warning FailedMount pod/spark-pi-d95nid148los7b6gcjmg-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-654a999f36d1a512-conf-map" not found 6m22s Warning FailedMount pod/spark-pi-d95nid148los7b6gcjmg-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d95nid148los7b6gcjmg-2bfcea9f36d193e3-driver-podspec-conf-map" not found 5m2s Warning FailedMount pod/scheduled-spark-pi-d95niph48los7b6gcjn0-1783331204001048776-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-b056b79f36d2dbfd-conf-map" not found 5m2s Warning FailedMount pod/scheduled-spark-pi-d95niph48los7b6gcjn0-1783331204001048776-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d95niph48los7b6gcjn0-1783331204001048776-453e229f36d2c8df-driver-podspec-conf-map" not found 3m43s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 2m19s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 75m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 74m Warning Deleting modelregistry/default-db-registry-1783326854338 Custom Resource default-db-registry-1783326854338 is being deleted from the namespace rhoai-model-registries 74m Warning Deleting modelregistry/default-db-registry-1783326854338 Custom Resource default-db-registry-1783326854338 is being deleted from the namespace rhoai-model-registries 75m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 164m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 163m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 163m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 69m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 69m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 75m Warning Deleting modelregistry/postgres-registry-1783326854338 Custom Resource postgres-registry-1783326854338 is being deleted from the namespace rhoai-model-registries 75m Warning Deleting modelregistry/postgres-registry-1783326854338 Custom Resource postgres-registry-1783326854338 is being deleted from the namespace rhoai-model-registries 164m Warning FailedMount pod/model-registry0-5bb8895ffc-xmmpg MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "model-registry0-kube-rbac-proxy" not found 164m Warning FailedMount pod/model-registry0-5bb8895ffc-xmmpg MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "model-registry0-kube-rbac-proxy" not found 127m Warning Unhealthy pod/model-catalog-postgres-686d85b578-6dx9x Readiness probe errored: rpc error: code = NotFound desc = container is not created or running: checking if PID of 2ad15be9ad4322d216897120a7898c54ffcb2451fdd03320e2798c386115ce14 is running failed: container process not found 126m Warning FailedMount pod/model-catalog-7cc8755dc9-m7h56 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 126m Warning Unhealthy pod/model-catalog-7cc8755dc9-m7h56 Readiness probe failed: dial tcp 10.130.3.44:8080: connect: connection refused 115m Warning FailedMount pod/model-catalog-7cc8755dc9-qtf2c MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 114m Warning Unhealthy pod/model-catalog-7cc8755dc9-qtf2c Readiness probe failed: dial tcp 10.129.3.177:8080: connect: connection refused 112m Warning FailedToUpdateEndpointSlices service/model-catalog Error updating Endpoint Slices for Service rhoai-model-registries/model-catalog: failed to update model-catalog-746vv EndpointSlice for Service rhoai-model-registries/model-catalog: Operation cannot be fulfilled on endpointslices.discovery.k8s.io "model-catalog-746vv": StorageError: invalid object, Code: 4, Key: /kubernetes.io/endpointslices/rhoai-model-registries/model-catalog-746vv, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 88277b4d-0133-42a3-b985-ce9b577646ca, UID in object meta: 100m Warning FailedMount pod/model-catalog-7cc8755dc9-dhk4t MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 100m Warning Unhealthy pod/model-catalog-7cc8755dc9-dhk4t Readiness probe failed: dial tcp 10.130.3.84:8080: connect: connection refused 77m Warning Unhealthy pod/mysql-db-registry-1783326854338-988474489-4qfs7 Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 76m Warning FailedMount pod/e2e-test-registry-56b8557c54-xmmgb MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "e2e-test-registry-kube-rbac-proxy-config" not found 74m Warning FailedMount pod/default-db-registry-1783326854338-5ff7dff967-dvscg MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "default-db-registry-1783326854338-kube-rbac-proxy" not found 74m Warning FailedMount pod/default-db-registry-1783326854338-5ff7dff967-dvscg MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "default-db-registry-1783326854338-kube-rbac-proxy" not found 74m Warning FailedMount pod/default-db-registry-1783326854338-5ff7dff967-dvscg MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "default-db-registry-1783326854338-kube-rbac-proxy-config" not found 74m Warning Unhealthy pod/default-db-registry-1783326854338-5ff7dff967-dvscg Startup probe failed: HTTP probe failed with statuscode: 503 73m Warning Unhealthy pod/model-registry-db-662874-7956bc68d9-xnssh Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 73m Warning FailedMount pod/test-model-registry-662874-748fdd9884-c2z4k MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "test-model-registry-662874-kube-rbac-proxy" not found 73m Warning FailedMount pod/test-model-registry-662874-748fdd9884-c2z4k MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "test-model-registry-662874-kube-rbac-proxy-config" not found 73m Warning FailedMount pod/test-model-registry-662874-748fdd9884-c2z4k MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "test-model-registry-662874-kube-rbac-proxy" not found {"level":"error","ts":"2026-07-06T09:45:02Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:45:48Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:46:22Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:47:05Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:47:41Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:48:41Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:49:20Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:50:09Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:50:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:51:35Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-07-06T09:51:57Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.19.18 Kubernetes Version: v1.32.9 --- Namespaces --- namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.0 Authorino Operator 1.4.0 authorino-operator.v1.3.1 Succeeded cert-manager-operator.v1.19.0 cert-manager Operator for Red Hat OpenShift 1.19.0 cert-manager-operator.v1.18.1 Succeeded custom-metrics-autoscaler.v2.19.0-1 Custom Metrics Autoscaler 2.19.0-1 custom-metrics-autoscaler.v2.18.1-2 Succeeded dns-operator.v1.4.0 DNS Operator 1.4.0 dns-operator.v1.3.1 Succeeded kueue-operator.v1.3.1 Red Hat build of Kueue 1.3.1 Succeeded limitador-operator.v1.4.0 Limitador Operator 1.4.0 limitador-operator.v1.3.1 Succeeded rhcl-operator.v1.4.0 Red Hat Connectivity Link 1.4.0 rhcl-operator.v1.3.4 Succeeded rhods-operator.3.5.0-ea.2 Red Hat OpenShift AI 3.5.0-ea.2 Succeeded servicemeshoperator3.v3.3.4 Red Hat OpenShift Service Mesh 3 3.3.4 servicemeshoperator3.v3.1.0 Succeeded --- CSVs not in Succeeded phase --- (all CSVs report phase Succeeded) --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aipipelines: managementState=Managed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Unmanaged llamastackoperator: managementState=n/a mlflowoperator: managementState=Managed modelregistry: managementState=Managed ogx: managementState=Managed ray: managementState=Managed sparkoperator: managementState=Managed trainer: managementState=Managed trainingoperator: managementState=Managed trustyai: managementState=Managed workbenches: managementState=Managed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= AIPipelinesReady: status=True reason= message= DashboardReady: status=True reason= message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=True reason= message= MLflowOperatorReady: status=True reason= message= MaaSPrerequisitesAvailable: status=False reason=MaaSPrerequisitesNotMet message=maas-default-gateway is missing required annotations: security.opendatahub.io/authorino-tls-bootstrap="true" ModelControllerReady: status=True reason= message= ModelRegistryReady: status=True reason= message= ModelsAsServiceReady: status=True reason=Reconciled message=MaaS platform manifests applied and maas-api deployment is available OGXReady: status=True reason= message= RayReady: status=True reason= message= SparkOperatorReady: status=True reason= message= TrainerReady: status=True reason= message= TrainingOperatorReady: status=True reason= message= TrustyAIReady: status=True reason= message= WorkbenchesReady: status=True reason= message= --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 6d20h Ready 2026-06-29T13:43:18Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=True reason=Ready message=Monitoring stack is initialized Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in DSCI CR OpenTelemetryCollectorAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Metrics not configured in DSCI CR Traces not configured in DSCI CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in DSCI CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR PersesAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR --- Component CRs (Ready) --- NAME READY REASON URL dashboard.components.platform.opendatahub.io/default-dashboard True NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON kueue.components.platform.opendatahub.io/default-kueue True NAME READY REASON mlflowoperator.components.platform.opendatahub.io/default-mlflowoperator True NAME READY REASON URI modelcontroller.components.platform.opendatahub.io/default-modelcontroller True NAME READY REASON modelregistry.components.platform.opendatahub.io/default-modelregistry True NAME READY REASON modelsasservice.components.platform.opendatahub.io/default-modelsasservice True NAME READY REASON ogx.components.platform.opendatahub.io/default-ogx True NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON sparkoperator.components.platform.opendatahub.io/default-sparkoperator True NAME READY REASON trainer.components.platform.opendatahub.io/default-trainer True NAME READY REASON trainingoperator.components.platform.opendatahub.io/default-trainingoperator True NAME READY REASON trustyai.components.platform.opendatahub.io/default-trustyai True NAME READY REASON workbenches.components.platform.opendatahub.io/default-workbenches True --- Deployments not fully ready (ODH/RHOAI namespaces) --- (all deployments report ready replicas == desired) --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- (all pods Running or Succeeded) --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 131m Warning FailedToRetrieveImagePullSecret pod/maas-controller-656b6867cb-5b9h9 Unable to retrieve some image pull secrets (maas-controller-dockercfg-7z2ct); attempting to pull the image may not succeed. 127m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 127m Warning FailedCreate replicaset/spark-operator-controller-57c5867998 Error creating: pods "spark-operator-controller-57c5867998-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 127m Warning BackOff pod/spark-operator-controller-57c5867998-szrpd Back-off restarting failed container controller in pod spark-operator-controller-57c5867998-szrpd_redhat-ods-applications(17ccf8a3-035b-4071-866a-0326e08f40ad) 127m Warning FailedToRetrieveImagePullSecret pod/spark-operator-controller-57c5867998-szrpd Unable to retrieve some image pull secrets (spark-operator-controller-dockercfg-czkf7); attempting to pull the image may not succeed. 127m Warning FailedMount pod/spark-operator-webhook-56c5bb987f-kswlc MountVolume.SetUp failed for volume "kube-api-access-2xwh5" : failed to fetch token: serviceaccounts "spark-operator-webhook" not found 126m Warning FailedMount pod/model-registry-operator-controller-manager-5f68f4fd76-d2jqr MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 126m Warning FailedMount pod/odh-model-controller-6b647f87c-gbfdf MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 126m Warning FailedCreate replicaset/kserve-controller-manager-6c8cb86d7f Error creating: pods "kserve-controller-manager-6c8cb86d7f-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 126m Warning FailedCreate replicaset/spark-operator-controller-57c5867998 Error creating: pods "spark-operator-controller-57c5867998-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 126m Warning FailedCreate replicaset/llmisvc-controller-manager-586c85c465 Error creating: pods "llmisvc-controller-manager-586c85c465-" is forbidden: error looking up service account redhat-ods-applications/llmisvc-controller-manager: serviceaccount "llmisvc-controller-manager" not found 126m Warning FailedMount pod/ogx-k8s-operator-controller-manager-67866ffc79-glpwq MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 126m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 126m Warning FailedMount pod/odh-notebook-controller-manager-7f6b58f576-gskzh MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 126m Warning FailedCreate replicaset/notebook-controller-deployment-5d7d79cf5b Error creating: pods "notebook-controller-deployment-5d7d79cf5b-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 126m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 125m Warning CalculateExpectedPodCountFailed poddisruptionbudget/ogx-k8s-operator-controller-manager-pdb Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-67866ffc79-glpwq" 125m Warning CalculateExpectedPodCountFailed poddisruptionbudget/ogx-k8s-operator-controller-manager-pdb Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-67866ffc79-rdjtg" 125m Warning Unhealthy pod/ogx-k8s-operator-controller-manager-67866ffc79-glpwq Readiness probe failed: Get "http://10.130.3.41:8081/readyz": dial tcp 10.130.3.41:8081: connect: connection refused 125m Warning FailedMount pod/spark-pi-d95lqfp48los74uh76b0-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-2a36089f36647321-conf-map" not found 125m Warning FailedMount pod/spark-pi-d95lqfp48los74uh76b0-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d95lqfp48los74uh76b0-77655d9f36645fa1-driver-podspec-conf-map" not found 125m Warning FailedToUpdateEndpoint endpoints/data-science-pipelines-operator-service Failed to update endpoint redhat-ods-applications/data-science-pipelines-operator-service: Operation cannot be fulfilled on endpoints "data-science-pipelines-operator-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/data-science-pipelines-operator-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 96541ebd-3e08-4f82-932c-ccc33d6fcb27, UID in object meta: 125m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-7fc7f76n57jb MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 125m Warning FailedMount pod/kuberay-operator-6b449f854b-v9wcc MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 125m Warning FailedCreate replicaset/rhods-dashboard-6b5cc9785c Error creating: pods "rhods-dashboard-6b5cc9785c-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 124m Warning FailedCreate replicaset/rhods-dashboard-6b5cc9785c Error creating: pods "rhods-dashboard-6b5cc9785c-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 124m Warning FailedMount pod/rhods-dashboard-6b5cc9785c-v7hkz MountVolume.SetUp failed for volume "modules-sa-token" : references non-existent secret key: token 124m Warning Failed pod/rhods-dashboard-6b5cc9785c-v7hkz Error: ErrImagePull 124m Warning Failed pod/rhods-dashboard-6b5cc9785c-v7hkz Failed to pull image "registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:88aa0c68b57cc272713ec1f909559adbeec11e509457c3d4843a49797567261b": initializing source docker://registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:88aa0c68b57cc272713ec1f909559adbeec11e509457c3d4843a49797567261b: (Mirrors also failed: [brew.registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:88aa0c68b57cc272713ec1f909559adbeec11e509457c3d4843a49797567261b: reading manifest sha256:88aa0c68b57cc272713ec1f909559adbeec11e509457c3d4843a49797567261b in brew.registry.redhat.io/rhoai/odh-mod-arch-gen-ai-rhel9: name unknown: Digest not found]... 124m Warning Failed pod/rhods-dashboard-6b5cc9785c-v7hkz Error: ImagePullBackOff 124m Warning FailedMount pod/scheduled-spark-pi-d95lqsh48los74uh76bg-1783324048001419356-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d95lqsh48los74uh76bg-1783324048001419356-1fb70c9f3665975b-driver-podspec-conf-map" not found 124m Warning FailedMount pod/scheduled-spark-pi-d95lqsh48los74uh76bg-1783324048001419356-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-6712f59f3665aa0d-conf-map" not found 124m Warning FailedToUpdateEndpointSlices service/data-science-pipelines-operator-service Error updating Endpoint Slices for Service redhat-ods-applications/data-science-pipelines-operator-service: failed to delete data-science-pipelines-operator-service-84mtw EndpointSlice for Service redhat-ods-applications/data-science-pipelines-operator-service: endpointslices.discovery.k8s.io "data-science-pipelines-operator-service-84mtw" not found 115m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 115m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 115m Warning InvalidConfiguration servicemonitor/workload-variant-autoscaler-controller-manager-metrics-monitor ServiceMonitor workload-variant-autoscaler-controller-manager-metrics-monitor was rejected due to invalid configuration: failed to get authorization token of type "": unable to get secret "workload-variant-autoscaler-metrics-reader-token": secrets "workload-variant-autoscaler-metrics-reader-token" not found 112m Warning FailedCreate replicaset/kserve-controller-manager-6c8cb86d7f Error creating: pods "kserve-controller-manager-6c8cb86d7f-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 109m Warning FailedMount pod/odh-model-controller-54bf759f64-kjt65 MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 109m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 108m Warning FailedMount pod/mlflow-operator-controller-manager-8646f94cf4-dsp7p MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 108m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 108m Warning FailedMount pod/maas-api-84c4b8dcb6-gb94r MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-serving-cert" not found 107m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-6f7f88c874 Error creating: pods "trustyai-service-operator-controller-manager-6f7f88c874-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 107m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning FailedMount pod/odh-model-controller-54bf759f64-22kmw MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 105m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 104m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 104m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning FailedToRetrieveImagePullSecret pod/llmisvc-controller-manager-586c85c465-hcdk8 Unable to retrieve some image pull secrets (llmisvc-controller-manager-dockercfg-x2zz7); attempting to pull the image may not succeed. 103m Warning FailedToRetrieveImagePullSecret pod/kserve-controller-manager-6c8cb86d7f-8ngts Unable to retrieve some image pull secrets (kserve-controller-manager-dockercfg-7zr65); attempting to pull the image may not succeed. 103m Warning FailedMount pod/odh-model-controller-54bf759f64-76pvs MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 103m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 103m Warning FailedMount pod/model-serving-api-78bbbb9bf8-4xzmv MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning FailedCreate replicaset/model-serving-api-78bbbb9bf8 Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 100m Warning FailedMount pod/model-registry-operator-controller-manager-5f68f4fd76-skcfc MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 100m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-7fc7f768f5wq MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 100m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-6f7f88c874 Error creating: pods "trustyai-service-operator-controller-manager-6f7f88c874-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 100m Warning FailedMount pod/mlflow-operator-controller-manager-8646f94cf4-pmz5p MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 100m Warning FailedMount pod/ogx-k8s-operator-controller-manager-67866ffc79-sjnxz MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 100m Warning FailedMount pod/odh-notebook-controller-manager-7f6b58f576-pwk7q MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 100m Warning FailedCreate replicaset/spark-operator-controller-57c5867998 Error creating: pods "spark-operator-controller-57c5867998-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 100m Warning FailedCreate replicaset/notebook-controller-deployment-5d7d79cf5b Error creating: pods "notebook-controller-deployment-5d7d79cf5b-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 100m Warning ErrorUpdatingResource pod/notebook-controller-deployment-5d7d79cf5b-gb4cr addLogicalPort failed for redhat-ods-applications/notebook-controller-deployment-5d7d79cf5b-gb4cr: failed to update pod redhat-ods-applications/notebook-controller-deployment-5d7d79cf5b-gb4cr: Operation cannot be fulfilled on pods "notebook-controller-deployment-5d7d79cf5b-gb4cr": the object has been modified; please apply your changes to the latest version and try again 100m Warning BackOff pod/trustyai-service-operator-controller-manager-6f7f88c874-5gk7q Back-off restarting failed container manager in pod trustyai-service-operator-controller-manager-6f7f88c874-5gk7q_redhat-ods-applications(afa97808-57db-46d5-8a99-ecb3ec30b632) 100m Warning FailedCreate replicaset/rhods-dashboard-6b5cc9785c Error creating: pods "rhods-dashboard-6b5cc9785c-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 100m Warning FailedMount pod/rhods-dashboard-6b5cc9785c-mpn5c MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 100m Warning FailedMount pod/rhods-dashboard-6b5cc9785c-2qxdw MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 100m Warning FailedMount pod/kuberay-operator-6b449f854b-mxlmx MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 99m Warning FailedCreate replicaset/maas-api-84c4b8dcb6 Error creating: pods "maas-api-84c4b8dcb6-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 96m Warning Unhealthy pod/mlflow-668464596d-sxm9n Readiness probe failed: Get "https://10.131.2.162:8443/mlflow/health": dial tcp 10.131.2.162:8443: connect: connection refused 94m Warning Unhealthy pod/mlflow-58f77fb56c-rpv9n Readiness probe failed: Get "https://10.131.2.165:8443/mlflow/health": dial tcp 10.131.2.165:8443: connect: connection refused 6m22s Warning FailedMount pod/spark-pi-d95nid148los7b6gcjmg-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-654a999f36d1a512-conf-map" not found 6m22s Warning FailedMount pod/spark-pi-d95nid148los7b6gcjmg-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d95nid148los7b6gcjmg-2bfcea9f36d193e3-driver-podspec-conf-map" not found 5m2s Warning FailedMount pod/scheduled-spark-pi-d95niph48los7b6gcjn0-1783331204001048776-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-b056b79f36d2dbfd-conf-map" not found 5m2s Warning FailedMount pod/scheduled-spark-pi-d95niph48los7b6gcjn0-1783331204001048776-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d95niph48los7b6gcjn0-1783331204001048776-453e229f36d2c8df-driver-podspec-conf-map" not found 3m43s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 2m19s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- (no events) --- namespace/rhoai-model-registries --- LAST SEEN TYPE REASON OBJECT MESSAGE 75m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 74m Warning Deleting modelregistry/default-db-registry-1783326854338 Custom Resource default-db-registry-1783326854338 is being deleted from the namespace rhoai-model-registries 74m Warning Deleting modelregistry/default-db-registry-1783326854338 Custom Resource default-db-registry-1783326854338 is being deleted from the namespace rhoai-model-registries 75m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 164m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 163m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 163m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 69m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 69m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 75m Warning Deleting modelregistry/postgres-registry-1783326854338 Custom Resource postgres-registry-1783326854338 is being deleted from the namespace rhoai-model-registries 75m Warning Deleting modelregistry/postgres-registry-1783326854338 Custom Resource postgres-registry-1783326854338 is being deleted from the namespace rhoai-model-registries 164m Warning FailedMount pod/model-registry0-5bb8895ffc-xmmpg MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "model-registry0-kube-rbac-proxy" not found 164m Warning FailedMount pod/model-registry0-5bb8895ffc-xmmpg MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "model-registry0-kube-rbac-proxy" not found 127m Warning Unhealthy pod/model-catalog-postgres-686d85b578-6dx9x Readiness probe errored: rpc error: code = NotFound desc = container is not created or running: checking if PID of 2ad15be9ad4322d216897120a7898c54ffcb2451fdd03320e2798c386115ce14 is running failed: container process not found 126m Warning FailedMount pod/model-catalog-7cc8755dc9-m7h56 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 126m Warning Unhealthy pod/model-catalog-7cc8755dc9-m7h56 Readiness probe failed: dial tcp 10.130.3.44:8080: connect: connection refused 115m Warning FailedMount pod/model-catalog-7cc8755dc9-qtf2c MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 114m Warning Unhealthy pod/model-catalog-7cc8755dc9-qtf2c Readiness probe failed: dial tcp 10.129.3.177:8080: connect: connection refused 112m Warning FailedToUpdateEndpointSlices service/model-catalog Error updating Endpoint Slices for Service rhoai-model-registries/model-catalog: failed to update model-catalog-746vv EndpointSlice for Service rhoai-model-registries/model-catalog: Operation cannot be fulfilled on endpointslices.discovery.k8s.io "model-catalog-746vv": StorageError: invalid object, Code: 4, Key: /kubernetes.io/endpointslices/rhoai-model-registries/model-catalog-746vv, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 88277b4d-0133-42a3-b985-ce9b577646ca, UID in object meta: 100m Warning FailedMount pod/model-catalog-7cc8755dc9-dhk4t MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 100m Warning Unhealthy pod/model-catalog-7cc8755dc9-dhk4t Readiness probe failed: dial tcp 10.130.3.84:8080: connect: connection refused 77m Warning Unhealthy pod/mysql-db-registry-1783326854338-988474489-4qfs7 Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 76m Warning FailedMount pod/e2e-test-registry-56b8557c54-xmmgb MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "e2e-test-registry-kube-rbac-proxy-config" not found 74m Warning FailedMount pod/default-db-registry-1783326854338-5ff7dff967-dvscg MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "default-db-registry-1783326854338-kube-rbac-proxy" not found 74m Warning FailedMount pod/default-db-registry-1783326854338-5ff7dff967-dvscg MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "default-db-registry-1783326854338-kube-rbac-proxy" not found 74m Warning FailedMount pod/default-db-registry-1783326854338-5ff7dff967-dvscg MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "default-db-registry-1783326854338-kube-rbac-proxy-config" not found 74m Warning Unhealthy pod/default-db-registry-1783326854338-5ff7dff967-dvscg Startup probe failed: HTTP probe failed with statuscode: 503 73m Warning Unhealthy pod/model-registry-db-662874-7956bc68d9-xnssh Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 73m Warning FailedMount pod/test-model-registry-662874-748fdd9884-c2z4k MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "test-model-registry-662874-kube-rbac-proxy" not found 73m Warning FailedMount pod/test-model-registry-662874-748fdd9884-c2z4k MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "test-model-registry-662874-kube-rbac-proxy-config" not found 73m Warning FailedMount pod/test-model-registry-662874-748fdd9884-c2z4k MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "test-model-registry-662874-kube-rbac-proxy" not found --- namespace/rhods-notebooks --- LAST SEEN TYPE REASON OBJECT MESSAGE --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-07-06T06:45:51Z) --- redhat-ods-operator/rhods-operator-6f6bb89fdf-59gvk --- {"level":"error","ts":"2026-07-06T09:45:02Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:45:48Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:46:22Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:47:05Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:47:41Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:48:41Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:49:20Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:50:09Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:50:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:51:35Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-6f6bb89fdf-59gvk.log ==================== {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"64e9863f-92f1-434b-afba-069a203a2247","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2d0cdfd4-3ef8-4875-b8a4-1c4334469f10"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2d0cdfd4-3ef8-4875-b8a4-1c4334469f10"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2d0cdfd4-3ef8-4875-b8a4-1c4334469f10","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2d0cdfd4-3ef8-4875-b8a4-1c4334469f10","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2d0cdfd4-3ef8-4875-b8a4-1c4334469f10","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:44:27Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2d0cdfd4-3ef8-4875-b8a4-1c4334469f10","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2d0cdfd4-3ef8-4875-b8a4-1c4334469f10","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2d0cdfd4-3ef8-4875-b8a4-1c4334469f10","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:27Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2c03fd53-e439-4785-aa14-76d9b73f18b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:28Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"62736be5-1765-43c3-89f3-ce5cf9db3abf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2d0cdfd4-3ef8-4875-b8a4-1c4334469f10","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:28Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"7f6336aa-83c4-4965-b4dc-277f35277b93","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:44:28Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"7f6336aa-83c4-4965-b4dc-277f35277b93","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:44:28Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"c90880f8-6c15-4d72-9496-724b8730dbd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:28Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"c90880f8-6c15-4d72-9496-724b8730dbd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:28Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"c90880f8-6c15-4d72-9496-724b8730dbd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/imagestreams.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:28Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"c90880f8-6c15-4d72-9496-724b8730dbd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.updateStatus"} {"level":"info","ts":"2026-07-06T09:44:28Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"c90880f8-6c15-4d72-9496-724b8730dbd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"2d0cdfd4-3ef8-4875-b8a4-1c4334469f10","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ab771a3f-1224-47e4-9f9e-af6c97cafcfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-06T09:44:29Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ab771a3f-1224-47e4-9f9e-af6c97cafcfc","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ab771a3f-1224-47e4-9f9e-af6c97cafcfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"dc8100b9-05c2-44a6-915e-846ac1eef045","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.reconcileModelCache"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"dc8100b9-05c2-44a6-915e-846ac1eef045","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"dc8100b9-05c2-44a6-915e-846ac1eef045","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"dc8100b9-05c2-44a6-915e-846ac1eef045","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"da5f1996-75e5-4356-8483-94fa0157d5ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:29Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"62736be5-1765-43c3-89f3-ce5cf9db3abf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"62736be5-1765-43c3-89f3-ce5cf9db3abf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"bc0baa0e-e92b-4cbf-9595-5175f6abc45d","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"bc0baa0e-e92b-4cbf-9595-5175f6abc45d","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"d00c1939-7b95-485f-a0d0-83d96533ce6d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"d00c1939-7b95-485f-a0d0-83d96533ce6d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.reconcileHardwareProfiles"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"d00c1939-7b95-485f-a0d0-83d96533ce6d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.updateStatus"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"d00c1939-7b95-485f-a0d0-83d96533ce6d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"d00c1939-7b95-485f-a0d0-83d96533ce6d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T09:44:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"cd962045-234e-442a-aa0b-9c01fd26e080","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.reconcileModelCache"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"045f39cd-8057-4bcd-a508-631a28ee1f0c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ab771a3f-1224-47e4-9f9e-af6c97cafcfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ab771a3f-1224-47e4-9f9e-af6c97cafcfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ab771a3f-1224-47e4-9f9e-af6c97cafcfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ab771a3f-1224-47e4-9f9e-af6c97cafcfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ab771a3f-1224-47e4-9f9e-af6c97cafcfc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"435e4a85-d1ef-4fac-91e5-4d59e7baef1d"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"435e4a85-d1ef-4fac-91e5-4d59e7baef1d"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"435e4a85-d1ef-4fac-91e5-4d59e7baef1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"435e4a85-d1ef-4fac-91e5-4d59e7baef1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"435e4a85-d1ef-4fac-91e5-4d59e7baef1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:44:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"435e4a85-d1ef-4fac-91e5-4d59e7baef1d","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"435e4a85-d1ef-4fac-91e5-4d59e7baef1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"435e4a85-d1ef-4fac-91e5-4d59e7baef1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"435e4a85-d1ef-4fac-91e5-4d59e7baef1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"435e4a85-d1ef-4fac-91e5-4d59e7baef1d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"c90880f8-6c15-4d72-9496-724b8730dbd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"62736be5-1765-43c3-89f3-ce5cf9db3abf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0ee135da-e7d7-4267-8308-6a39bf02fa09"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0ee135da-e7d7-4267-8308-6a39bf02fa09"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0ee135da-e7d7-4267-8308-6a39bf02fa09","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0ee135da-e7d7-4267-8308-6a39bf02fa09","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0ee135da-e7d7-4267-8308-6a39bf02fa09","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e6afc07-95d5-4dfa-850c-5a6b3b66df4d"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e6afc07-95d5-4dfa-850c-5a6b3b66df4d"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e6afc07-95d5-4dfa-850c-5a6b3b66df4d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e6afc07-95d5-4dfa-850c-5a6b3b66df4d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e6afc07-95d5-4dfa-850c-5a6b3b66df4d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:44:30Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e6afc07-95d5-4dfa-850c-5a6b3b66df4d","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e6afc07-95d5-4dfa-850c-5a6b3b66df4d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e6afc07-95d5-4dfa-850c-5a6b3b66df4d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0ee135da-e7d7-4267-8308-6a39bf02fa09","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:30Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e6afc07-95d5-4dfa-850c-5a6b3b66df4d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0ee135da-e7d7-4267-8308-6a39bf02fa09","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0ee135da-e7d7-4267-8308-6a39bf02fa09","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0ee135da-e7d7-4267-8308-6a39bf02fa09","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e6afc07-95d5-4dfa-850c-5a6b3b66df4d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c8778195-1dcb-4a95-ae3b-c099b4f8a6e4"} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c8778195-1dcb-4a95-ae3b-c099b4f8a6e4"} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c8778195-1dcb-4a95-ae3b-c099b4f8a6e4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c8778195-1dcb-4a95-ae3b-c099b4f8a6e4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c8778195-1dcb-4a95-ae3b-c099b4f8a6e4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:44:31Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c8778195-1dcb-4a95-ae3b-c099b4f8a6e4","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c8778195-1dcb-4a95-ae3b-c099b4f8a6e4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c8778195-1dcb-4a95-ae3b-c099b4f8a6e4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c8778195-1dcb-4a95-ae3b-c099b4f8a6e4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:31Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c8778195-1dcb-4a95-ae3b-c099b4f8a6e4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d0733517-089d-4be1-946f-59bc279297a1"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d0733517-089d-4be1-946f-59bc279297a1"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d0733517-089d-4be1-946f-59bc279297a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d0733517-089d-4be1-946f-59bc279297a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d0733517-089d-4be1-946f-59bc279297a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d0733517-089d-4be1-946f-59bc279297a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d0733517-089d-4be1-946f-59bc279297a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d0733517-089d-4be1-946f-59bc279297a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d0733517-089d-4be1-946f-59bc279297a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0fda2e42-a088-4ebe-8ae5-583293c05b2c"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0fda2e42-a088-4ebe-8ae5-583293c05b2c"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0fda2e42-a088-4ebe-8ae5-583293c05b2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0fda2e42-a088-4ebe-8ae5-583293c05b2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0fda2e42-a088-4ebe-8ae5-583293c05b2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:44:41Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0fda2e42-a088-4ebe-8ae5-583293c05b2c","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0fda2e42-a088-4ebe-8ae5-583293c05b2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0fda2e42-a088-4ebe-8ae5-583293c05b2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0fda2e42-a088-4ebe-8ae5-583293c05b2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0fda2e42-a088-4ebe-8ae5-583293c05b2c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e8a3c953-10f7-4215-8424-e03ea2b27fea"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e8a3c953-10f7-4215-8424-e03ea2b27fea"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e8a3c953-10f7-4215-8424-e03ea2b27fea","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e8a3c953-10f7-4215-8424-e03ea2b27fea","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e8a3c953-10f7-4215-8424-e03ea2b27fea","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e8a3c953-10f7-4215-8424-e03ea2b27fea","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e8a3c953-10f7-4215-8424-e03ea2b27fea","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e8a3c953-10f7-4215-8424-e03ea2b27fea","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e8a3c953-10f7-4215-8424-e03ea2b27fea","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f46d03ae-e60e-45e9-949c-8b163e403fde"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f46d03ae-e60e-45e9-949c-8b163e403fde"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f46d03ae-e60e-45e9-949c-8b163e403fde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f46d03ae-e60e-45e9-949c-8b163e403fde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f46d03ae-e60e-45e9-949c-8b163e403fde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:44:42Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f46d03ae-e60e-45e9-949c-8b163e403fde","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f46d03ae-e60e-45e9-949c-8b163e403fde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f46d03ae-e60e-45e9-949c-8b163e403fde","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f46d03ae-e60e-45e9-949c-8b163e403fde","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"f46d03ae-e60e-45e9-949c-8b163e403fde","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bb6c993-d9ae-4ff2-b54d-c7e19e3e1691","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:42Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"e80cf81c-065b-4122-b1a2-b1b20e8596c3","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:44:42Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"e80cf81c-065b-4122-b1a2-b1b20e8596c3","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5aad3682-603b-4966-8576-19e4d64b488c"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5aad3682-603b-4966-8576-19e4d64b488c"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5aad3682-603b-4966-8576-19e4d64b488c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5aad3682-603b-4966-8576-19e4d64b488c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5aad3682-603b-4966-8576-19e4d64b488c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5aad3682-603b-4966-8576-19e4d64b488c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5aad3682-603b-4966-8576-19e4d64b488c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5aad3682-603b-4966-8576-19e4d64b488c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5aad3682-603b-4966-8576-19e4d64b488c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0f63c6e3-d080-40b4-8683-b8aad036e05d"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0f63c6e3-d080-40b4-8683-b8aad036e05d"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0f63c6e3-d080-40b4-8683-b8aad036e05d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0f63c6e3-d080-40b4-8683-b8aad036e05d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0f63c6e3-d080-40b4-8683-b8aad036e05d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:44:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0f63c6e3-d080-40b4-8683-b8aad036e05d","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0f63c6e3-d080-40b4-8683-b8aad036e05d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0f63c6e3-d080-40b4-8683-b8aad036e05d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5dd0b819-6d55-44da-bb46-cfb88b19e0dd"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5dd0b819-6d55-44da-bb46-cfb88b19e0dd"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5dd0b819-6d55-44da-bb46-cfb88b19e0dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5dd0b819-6d55-44da-bb46-cfb88b19e0dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5dd0b819-6d55-44da-bb46-cfb88b19e0dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5dd0b819-6d55-44da-bb46-cfb88b19e0dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0f63c6e3-d080-40b4-8683-b8aad036e05d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5dd0b819-6d55-44da-bb46-cfb88b19e0dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5dd0b819-6d55-44da-bb46-cfb88b19e0dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"5dd0b819-6d55-44da-bb46-cfb88b19e0dd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0f63c6e3-d080-40b4-8683-b8aad036e05d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0848374b-a121-46d2-9809-fff4765b7e66"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0848374b-a121-46d2-9809-fff4765b7e66"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0848374b-a121-46d2-9809-fff4765b7e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0848374b-a121-46d2-9809-fff4765b7e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0848374b-a121-46d2-9809-fff4765b7e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:44:44Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0848374b-a121-46d2-9809-fff4765b7e66","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0848374b-a121-46d2-9809-fff4765b7e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0848374b-a121-46d2-9809-fff4765b7e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0848374b-a121-46d2-9809-fff4765b7e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0848374b-a121-46d2-9809-fff4765b7e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3fbeac28-1d93-4a80-bbb5-8491cf59a399","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:44Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"0dc0b376-dc98-4a8a-96bc-fb9285e0e613","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:44:44Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"0dc0b376-dc98-4a8a-96bc-fb9285e0e613","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:44:46Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c2010936-5d3b-4ead-bcf7-64007147f104"} {"level":"info","ts":"2026-07-06T09:44:46Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c2010936-5d3b-4ead-bcf7-64007147f104"} {"level":"info","ts":"2026-07-06T09:44:46Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c2010936-5d3b-4ead-bcf7-64007147f104","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:44:46Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c2010936-5d3b-4ead-bcf7-64007147f104","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:46Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c2010936-5d3b-4ead-bcf7-64007147f104","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:46Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c2010936-5d3b-4ead-bcf7-64007147f104","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c2010936-5d3b-4ead-bcf7-64007147f104","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c2010936-5d3b-4ead-bcf7-64007147f104","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c2010936-5d3b-4ead-bcf7-64007147f104","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"403a71d6-be99-4784-b74e-f35ea4383c1b"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"403a71d6-be99-4784-b74e-f35ea4383c1b"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"403a71d6-be99-4784-b74e-f35ea4383c1b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"403a71d6-be99-4784-b74e-f35ea4383c1b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"403a71d6-be99-4784-b74e-f35ea4383c1b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"403a71d6-be99-4784-b74e-f35ea4383c1b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8dc6d0-ac77-47b6-a0d0-b1802da52e59"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8dc6d0-ac77-47b6-a0d0-b1802da52e59"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8dc6d0-ac77-47b6-a0d0-b1802da52e59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8dc6d0-ac77-47b6-a0d0-b1802da52e59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8dc6d0-ac77-47b6-a0d0-b1802da52e59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:44:47Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8dc6d0-ac77-47b6-a0d0-b1802da52e59","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8dc6d0-ac77-47b6-a0d0-b1802da52e59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8dc6d0-ac77-47b6-a0d0-b1802da52e59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8dc6d0-ac77-47b6-a0d0-b1802da52e59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"403a71d6-be99-4784-b74e-f35ea4383c1b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"403a71d6-be99-4784-b74e-f35ea4383c1b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"403a71d6-be99-4784-b74e-f35ea4383c1b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"0c8dc6d0-ac77-47b6-a0d0-b1802da52e59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0a76c5f9-f5e6-4d5f-ae1f-1a9719e5ab84"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0a76c5f9-f5e6-4d5f-ae1f-1a9719e5ab84"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0a76c5f9-f5e6-4d5f-ae1f-1a9719e5ab84","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0a76c5f9-f5e6-4d5f-ae1f-1a9719e5ab84","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0a76c5f9-f5e6-4d5f-ae1f-1a9719e5ab84","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0a76c5f9-f5e6-4d5f-ae1f-1a9719e5ab84","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0a76c5f9-f5e6-4d5f-ae1f-1a9719e5ab84","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0a76c5f9-f5e6-4d5f-ae1f-1a9719e5ab84","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0a76c5f9-f5e6-4d5f-ae1f-1a9719e5ab84","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"116cb6fc-fae5-4b81-9a8d-a6704bea6503"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"116cb6fc-fae5-4b81-9a8d-a6704bea6503"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"116cb6fc-fae5-4b81-9a8d-a6704bea6503","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"116cb6fc-fae5-4b81-9a8d-a6704bea6503","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"116cb6fc-fae5-4b81-9a8d-a6704bea6503","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:44:56Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"116cb6fc-fae5-4b81-9a8d-a6704bea6503","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"116cb6fc-fae5-4b81-9a8d-a6704bea6503","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"116cb6fc-fae5-4b81-9a8d-a6704bea6503","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"116cb6fc-fae5-4b81-9a8d-a6704bea6503","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"116cb6fc-fae5-4b81-9a8d-a6704bea6503","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:44:56Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"07200dd3-5873-4b48-af3d-965723648a68","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:44:57Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"ecbd4295-b4fc-46ca-b3c0-46185af2281f","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:44:57Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"ecbd4295-b4fc-46ca-b3c0-46185af2281f","name":"redhat-ods-applications"} {"level":"error","ts":"2026-07-06T09:45:02Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ea17dfa1-6a39-4d8c-9b26-9648849cef59"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ea17dfa1-6a39-4d8c-9b26-9648849cef59"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ea17dfa1-6a39-4d8c-9b26-9648849cef59","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ea17dfa1-6a39-4d8c-9b26-9648849cef59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ea17dfa1-6a39-4d8c-9b26-9648849cef59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ea17dfa1-6a39-4d8c-9b26-9648849cef59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ea17dfa1-6a39-4d8c-9b26-9648849cef59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ea17dfa1-6a39-4d8c-9b26-9648849cef59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"ea17dfa1-6a39-4d8c-9b26-9648849cef59","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"744ef2a2-9a65-488e-8e54-b55c7719376c"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"744ef2a2-9a65-488e-8e54-b55c7719376c"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"744ef2a2-9a65-488e-8e54-b55c7719376c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"744ef2a2-9a65-488e-8e54-b55c7719376c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"744ef2a2-9a65-488e-8e54-b55c7719376c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e74048c-702d-43ae-bca6-56ee005230bb"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e74048c-702d-43ae-bca6-56ee005230bb"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e74048c-702d-43ae-bca6-56ee005230bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e74048c-702d-43ae-bca6-56ee005230bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e74048c-702d-43ae-bca6-56ee005230bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:45:04Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e74048c-702d-43ae-bca6-56ee005230bb","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e74048c-702d-43ae-bca6-56ee005230bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e74048c-702d-43ae-bca6-56ee005230bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"744ef2a2-9a65-488e-8e54-b55c7719376c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e74048c-702d-43ae-bca6-56ee005230bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"744ef2a2-9a65-488e-8e54-b55c7719376c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"744ef2a2-9a65-488e-8e54-b55c7719376c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"744ef2a2-9a65-488e-8e54-b55c7719376c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"9e74048c-702d-43ae-bca6-56ee005230bb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d46051ab-07d3-4c67-9e8e-1a511331a9a5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:45:04Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"d6bbbd51-0de6-4235-99aa-8a4e3886e2f7","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:45:04Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"d6bbbd51-0de6-4235-99aa-8a4e3886e2f7","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:45:10Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0d58c32d-405d-42db-97bc-7229dc07fe91"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0d58c32d-405d-42db-97bc-7229dc07fe91"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0d58c32d-405d-42db-97bc-7229dc07fe91","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0d58c32d-405d-42db-97bc-7229dc07fe91","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0d58c32d-405d-42db-97bc-7229dc07fe91","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0d58c32d-405d-42db-97bc-7229dc07fe91","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0d58c32d-405d-42db-97bc-7229dc07fe91","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0d58c32d-405d-42db-97bc-7229dc07fe91","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0d58c32d-405d-42db-97bc-7229dc07fe91","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b532056f-4753-426d-bc29-adc79115a829"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b532056f-4753-426d-bc29-adc79115a829"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b532056f-4753-426d-bc29-adc79115a829","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b532056f-4753-426d-bc29-adc79115a829","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b532056f-4753-426d-bc29-adc79115a829","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:45:16Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b532056f-4753-426d-bc29-adc79115a829","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b532056f-4753-426d-bc29-adc79115a829","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b532056f-4753-426d-bc29-adc79115a829","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b532056f-4753-426d-bc29-adc79115a829","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b532056f-4753-426d-bc29-adc79115a829","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"931e3771-bb62-4711-92cd-a17afe45356a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:45:17Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"2aa8b3ad-81e0-4ec4-aabb-0f26c153a6a1","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:45:17Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"2aa8b3ad-81e0-4ec4-aabb-0f26c153a6a1","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T09:45:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"7e1897dd-7ca0-4e22-84f5-9ed958db6858","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T09:45:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"8e0b3ee9-e3a4-4cd5-b458-8646a8024c29","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6c402886-d05b-4f46-b793-d3eec08f0d66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:45:38Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"93fda052-94a7-469e-8d48-6fd790aa7db4","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"93fda052-94a7-469e-8d48-6fd790aa7db4","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:38Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"74cb7f96-d528-4d0c-bdeb-5af4da118fb8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-06T09:45:48Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"af0aceac-5be7-4d98-b65a-774d8ee41cbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:45:49Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"052a54c9-549a-4359-ac2e-d478a690f8f1","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:45:49Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"052a54c9-549a-4359-ac2e-d478a690f8f1","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:45:49Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5"} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5"} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T09:45:50Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:45:50Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"a677e520-9169-4119-a781-35a6c6ad27f5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"ab03d965-c775-4c76-9e31-d354fcdae696","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:46:03Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"b261ef60-2735-40f4-ab21-a3f718552ad5","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:46:03Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"b261ef60-2735-40f4-ab21-a3f718552ad5","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:14Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"073e6224-f54b-4379-9482-28d64be164d1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:46:15Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"63143a34-ecd0-4a80-9615-0d0952fa710a","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:46:15Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"63143a34-ecd0-4a80-9615-0d0952fa710a","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T09:46:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"29f407b0-9322-4276-8f4f-f39970d2920b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-06T09:46:22Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c598d7e1-7788-4fe0-a884-50a5470823ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:46:58Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"e090557b-ebe5-4ad4-838c-1c36dede1d83","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"e090557b-ebe5-4ad4-838c-1c36dede1d83","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:46:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"63a51bb6-c725-4022-bd73-991ad7d93da8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-06T09:47:05Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"86466ce0-0cd7-4843-9c5d-36584fcca697","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:08Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"f1d81636-ad2c-4443-a13c-a3a44c39c87d","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:47:08Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"f1d81636-ad2c-4443-a13c-a3a44c39c87d","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"b7cbb07c-c9de-4117-bca0-e946a3309fbe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:22Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"5eb1c161-584c-4655-bfe4-4bcca75eec50","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:47:22Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"5eb1c161-584c-4655-bfe4-4bcca75eec50","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3038978a-fe39-4be1-aab5-a000ea56cd21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"51307812-9a93-48e4-8082-722034db5b21"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"51307812-9a93-48e4-8082-722034db5b21"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"51307812-9a93-48e4-8082-722034db5b21","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"51307812-9a93-48e4-8082-722034db5b21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"51307812-9a93-48e4-8082-722034db5b21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"51307812-9a93-48e4-8082-722034db5b21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"a37c883e-f242-46e3-9365-f3ccc7091d9d","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"a37c883e-f242-46e3-9365-f3ccc7091d9d","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"51307812-9a93-48e4-8082-722034db5b21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f712e040-3ee2-4f46-a300-cf739c2f607e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"51307812-9a93-48e4-8082-722034db5b21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"51307812-9a93-48e4-8082-722034db5b21","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"617b0fdb-3337-471d-83f5-d84b74143e3a"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"617b0fdb-3337-471d-83f5-d84b74143e3a"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"617b0fdb-3337-471d-83f5-d84b74143e3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"617b0fdb-3337-471d-83f5-d84b74143e3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"617b0fdb-3337-471d-83f5-d84b74143e3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"617b0fdb-3337-471d-83f5-d84b74143e3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"617b0fdb-3337-471d-83f5-d84b74143e3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"617b0fdb-3337-471d-83f5-d84b74143e3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:35Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"617b0fdb-3337-471d-83f5-d84b74143e3a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"2bbe7b24-660b-48e6-a5ec-3e7af7fd3b14","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:36Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"a442b24c-f98c-422b-9d18-e94d8bf52744","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:47:36Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"a442b24c-f98c-422b-9d18-e94d8bf52744","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"b3ec9c1e-bd21-494b-8500-2e898477c375"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"b3ec9c1e-bd21-494b-8500-2e898477c375"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"b3ec9c1e-bd21-494b-8500-2e898477c375","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"b3ec9c1e-bd21-494b-8500-2e898477c375","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"b3ec9c1e-bd21-494b-8500-2e898477c375","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"b3ec9c1e-bd21-494b-8500-2e898477c375","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"b3ec9c1e-bd21-494b-8500-2e898477c375","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"b3ec9c1e-bd21-494b-8500-2e898477c375","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"b3ec9c1e-bd21-494b-8500-2e898477c375","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"86613844-d037-46c8-a89e-1b490d97e2f2"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"86613844-d037-46c8-a89e-1b490d97e2f2"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"86613844-d037-46c8-a89e-1b490d97e2f2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"86613844-d037-46c8-a89e-1b490d97e2f2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"86613844-d037-46c8-a89e-1b490d97e2f2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"86613844-d037-46c8-a89e-1b490d97e2f2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"86613844-d037-46c8-a89e-1b490d97e2f2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"86613844-d037-46c8-a89e-1b490d97e2f2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:40Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"86613844-d037-46c8-a89e-1b490d97e2f2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-06T09:47:41Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"99430545-3062-49f1-a932-c1d885ccbf1e"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"99430545-3062-49f1-a932-c1d885ccbf1e"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"99430545-3062-49f1-a932-c1d885ccbf1e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"99430545-3062-49f1-a932-c1d885ccbf1e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"99430545-3062-49f1-a932-c1d885ccbf1e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"99430545-3062-49f1-a932-c1d885ccbf1e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"99430545-3062-49f1-a932-c1d885ccbf1e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"99430545-3062-49f1-a932-c1d885ccbf1e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"99430545-3062-49f1-a932-c1d885ccbf1e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"3d6a4636-0a52-4995-b805-96b1c652c40d"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"3d6a4636-0a52-4995-b805-96b1c652c40d"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"3d6a4636-0a52-4995-b805-96b1c652c40d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"3d6a4636-0a52-4995-b805-96b1c652c40d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"3d6a4636-0a52-4995-b805-96b1c652c40d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c641f98d-a296-4078-b05f-3275f1a5c4a8"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c641f98d-a296-4078-b05f-3275f1a5c4a8"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c641f98d-a296-4078-b05f-3275f1a5c4a8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c641f98d-a296-4078-b05f-3275f1a5c4a8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c641f98d-a296-4078-b05f-3275f1a5c4a8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:47:45Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c641f98d-a296-4078-b05f-3275f1a5c4a8","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c641f98d-a296-4078-b05f-3275f1a5c4a8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c641f98d-a296-4078-b05f-3275f1a5c4a8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"3d6a4636-0a52-4995-b805-96b1c652c40d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c641f98d-a296-4078-b05f-3275f1a5c4a8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"3d6a4636-0a52-4995-b805-96b1c652c40d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"3d6a4636-0a52-4995-b805-96b1c652c40d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"3d6a4636-0a52-4995-b805-96b1c652c40d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"63d2a129-0ee1-464d-9f37-86b346056f98"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"63d2a129-0ee1-464d-9f37-86b346056f98"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"63d2a129-0ee1-464d-9f37-86b346056f98","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"63d2a129-0ee1-464d-9f37-86b346056f98","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"63d2a129-0ee1-464d-9f37-86b346056f98","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"63d2a129-0ee1-464d-9f37-86b346056f98","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"63d2a129-0ee1-464d-9f37-86b346056f98","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"63d2a129-0ee1-464d-9f37-86b346056f98","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"63d2a129-0ee1-464d-9f37-86b346056f98","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"4b398e56-f155-4b96-9b7b-a64202d7ffbb"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"4b398e56-f155-4b96-9b7b-a64202d7ffbb"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"4b398e56-f155-4b96-9b7b-a64202d7ffbb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"4b398e56-f155-4b96-9b7b-a64202d7ffbb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"4b398e56-f155-4b96-9b7b-a64202d7ffbb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"4b398e56-f155-4b96-9b7b-a64202d7ffbb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"c641f98d-a296-4078-b05f-3275f1a5c4a8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1114661b-8d2e-4e09-a404-ecc15ab78429","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"4b398e56-f155-4b96-9b7b-a64202d7ffbb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"4b398e56-f155-4b96-9b7b-a64202d7ffbb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:45Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"4b398e56-f155-4b96-9b7b-a64202d7ffbb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:46Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"6b081e6c-fcd3-4030-8101-9ad7ce2cd35c","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:47:46Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"6b081e6c-fcd3-4030-8101-9ad7ce2cd35c","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d62a299a-9525-41be-bde7-1d7ca01bb68c"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d62a299a-9525-41be-bde7-1d7ca01bb68c"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d62a299a-9525-41be-bde7-1d7ca01bb68c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d62a299a-9525-41be-bde7-1d7ca01bb68c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d62a299a-9525-41be-bde7-1d7ca01bb68c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d62a299a-9525-41be-bde7-1d7ca01bb68c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d62a299a-9525-41be-bde7-1d7ca01bb68c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d62a299a-9525-41be-bde7-1d7ca01bb68c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d62a299a-9525-41be-bde7-1d7ca01bb68c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1e821191-074e-405b-ade1-4eb323ee71e2"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1e821191-074e-405b-ade1-4eb323ee71e2"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1e821191-074e-405b-ade1-4eb323ee71e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1e821191-074e-405b-ade1-4eb323ee71e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1e821191-074e-405b-ade1-4eb323ee71e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:47:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1e821191-074e-405b-ade1-4eb323ee71e2","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1e821191-074e-405b-ade1-4eb323ee71e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1e821191-074e-405b-ade1-4eb323ee71e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1e821191-074e-405b-ade1-4eb323ee71e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"1e821191-074e-405b-ade1-4eb323ee71e2","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:47:58Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"50c3e3a7-dba4-4bb3-8900-ec30981b833c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:47:59Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"486302ec-57a0-4c18-bab1-22a1765271c1","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:47:59Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"486302ec-57a0-4c18-bab1-22a1765271c1","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"72ea633c-fd27-4d1d-bf45-e599fb89ad8f"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"72ea633c-fd27-4d1d-bf45-e599fb89ad8f"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"72ea633c-fd27-4d1d-bf45-e599fb89ad8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"72ea633c-fd27-4d1d-bf45-e599fb89ad8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"72ea633c-fd27-4d1d-bf45-e599fb89ad8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"72ea633c-fd27-4d1d-bf45-e599fb89ad8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"72ea633c-fd27-4d1d-bf45-e599fb89ad8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"72ea633c-fd27-4d1d-bf45-e599fb89ad8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"72ea633c-fd27-4d1d-bf45-e599fb89ad8f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0c83274a-3e5a-4749-9c25-899ee2069b43"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0c83274a-3e5a-4749-9c25-899ee2069b43"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0c83274a-3e5a-4749-9c25-899ee2069b43","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0c83274a-3e5a-4749-9c25-899ee2069b43","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0c83274a-3e5a-4749-9c25-899ee2069b43","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6e732e10-dbe2-443c-8cc4-704c22a92963"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6e732e10-dbe2-443c-8cc4-704c22a92963"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6e732e10-dbe2-443c-8cc4-704c22a92963","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6e732e10-dbe2-443c-8cc4-704c22a92963","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6e732e10-dbe2-443c-8cc4-704c22a92963","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:48:00Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6e732e10-dbe2-443c-8cc4-704c22a92963","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6e732e10-dbe2-443c-8cc4-704c22a92963","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6e732e10-dbe2-443c-8cc4-704c22a92963","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0c83274a-3e5a-4749-9c25-899ee2069b43","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6e732e10-dbe2-443c-8cc4-704c22a92963","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0c83274a-3e5a-4749-9c25-899ee2069b43","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0c83274a-3e5a-4749-9c25-899ee2069b43","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0c83274a-3e5a-4749-9c25-899ee2069b43","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d6a53fcc-2ccd-4059-8228-b63240ecc838"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d6a53fcc-2ccd-4059-8228-b63240ecc838"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d6a53fcc-2ccd-4059-8228-b63240ecc838","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d6a53fcc-2ccd-4059-8228-b63240ecc838","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d6a53fcc-2ccd-4059-8228-b63240ecc838","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d6a53fcc-2ccd-4059-8228-b63240ecc838","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d6a53fcc-2ccd-4059-8228-b63240ecc838","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d6a53fcc-2ccd-4059-8228-b63240ecc838","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"d6a53fcc-2ccd-4059-8228-b63240ecc838","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"96100a1a-f877-4741-a12b-ebd8e0ffbbd5"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"96100a1a-f877-4741-a12b-ebd8e0ffbbd5"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"96100a1a-f877-4741-a12b-ebd8e0ffbbd5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"96100a1a-f877-4741-a12b-ebd8e0ffbbd5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"96100a1a-f877-4741-a12b-ebd8e0ffbbd5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"96100a1a-f877-4741-a12b-ebd8e0ffbbd5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6e732e10-dbe2-443c-8cc4-704c22a92963","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"96100a1a-f877-4741-a12b-ebd8e0ffbbd5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"96100a1a-f877-4741-a12b-ebd8e0ffbbd5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"96100a1a-f877-4741-a12b-ebd8e0ffbbd5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"57a8b5e5-d52e-43cb-92b5-ffb8be394de0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:00Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"bbe2b863-c30f-4559-ab5a-7a6de97d9693","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:48:00Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"bbe2b863-c30f-4559-ab5a-7a6de97d9693","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"1aee3aaa-87bb-4cff-b781-66fc66c99432"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"1aee3aaa-87bb-4cff-b781-66fc66c99432"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"1aee3aaa-87bb-4cff-b781-66fc66c99432","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"1aee3aaa-87bb-4cff-b781-66fc66c99432","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"1aee3aaa-87bb-4cff-b781-66fc66c99432","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"1aee3aaa-87bb-4cff-b781-66fc66c99432","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"1aee3aaa-87bb-4cff-b781-66fc66c99432","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"1aee3aaa-87bb-4cff-b781-66fc66c99432","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"1aee3aaa-87bb-4cff-b781-66fc66c99432","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecd621b9-c277-44b4-ba77-d63d452c1408"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecd621b9-c277-44b4-ba77-d63d452c1408"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecd621b9-c277-44b4-ba77-d63d452c1408","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecd621b9-c277-44b4-ba77-d63d452c1408","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecd621b9-c277-44b4-ba77-d63d452c1408","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:48:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecd621b9-c277-44b4-ba77-d63d452c1408","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecd621b9-c277-44b4-ba77-d63d452c1408","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecd621b9-c277-44b4-ba77-d63d452c1408","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecd621b9-c277-44b4-ba77-d63d452c1408","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ecd621b9-c277-44b4-ba77-d63d452c1408","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"460d8dde-075f-463b-b2ba-ba83562370b5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:13Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"28a29b8c-4c2c-496e-93ef-b60e1a80ab00","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:48:13Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"28a29b8c-4c2c-496e-93ef-b60e1a80ab00","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b5e3b98b-ddb0-406b-989a-24dd9bcf35fc"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b5e3b98b-ddb0-406b-989a-24dd9bcf35fc"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b5e3b98b-ddb0-406b-989a-24dd9bcf35fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b5e3b98b-ddb0-406b-989a-24dd9bcf35fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"reconcile","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"apply","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.initializeModules"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b5e3b98b-ddb0-406b-989a-24dd9bcf35fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e"} {"level":"info","ts":"2026-07-06T09:48:15Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b5e3b98b-ddb0-406b-989a-24dd9bcf35fc","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b5e3b98b-ddb0-406b-989a-24dd9bcf35fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b5e3b98b-ddb0-406b-989a-24dd9bcf35fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7e5359ae-097c-4382-b790-63c9ad869c8e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"98f10d57-f571-4b52-ab64-7480fd40ed16","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"reconcile","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"apply","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.initialize"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.customizeManifests"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.configureDependencies"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"reconcile","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"db610252-d789-4c8f-8068-b75fa25b0e89"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"apply","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"db610252-d789-4c8f-8068-b75fa25b0e89"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"db610252-d789-4c8f-8068-b75fa25b0e89","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainingoperator.initialize"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"db610252-d789-4c8f-8068-b75fa25b0e89","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"db610252-d789-4c8f-8068-b75fa25b0e89","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.updateStatus"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"679b5550-d8f1-4a0b-bbd5-344755d4173f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"db610252-d789-4c8f-8068-b75fa25b0e89","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"reconcile","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"eb7000ee-d49b-4e77-81fd-1489fa885858"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"apply","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"eb7000ee-d49b-4e77-81fd-1489fa885858"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"eb7000ee-d49b-4e77-81fd-1489fa885858","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/feastoperator.initialize"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"eb7000ee-d49b-4e77-81fd-1489fa885858","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/feastoperator.setKustomizedParams"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"eb7000ee-d49b-4e77-81fd-1489fa885858","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"eb7000ee-d49b-4e77-81fd-1489fa885858","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"eb7000ee-d49b-4e77-81fd-1489fa885858","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/feastoperator.migrateDeploymentSelector"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"eb7000ee-d49b-4e77-81fd-1489fa885858","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"db610252-d789-4c8f-8068-b75fa25b0e89","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"db610252-d789-4c8f-8068-b75fa25b0e89","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"db610252-d789-4c8f-8068-b75fa25b0e89","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"reconcile","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"20907ff2-e58a-4eb4-ad58-e5c3ba05849f"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"apply","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"20907ff2-e58a-4eb4-ad58-e5c3ba05849f"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"20907ff2-e58a-4eb4-ad58-e5c3ba05849f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trustyai.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:48:15Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"13eaaa8b-4170-4125-8176-2f0100528a21","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"13eaaa8b-4170-4125-8176-2f0100528a21","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"7a83b820-6990-40dd-9143-9a145dc59cf4"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"7a83b820-6990-40dd-9143-9a145dc59cf4"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"7a83b820-6990-40dd-9143-9a145dc59cf4","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"7a83b820-6990-40dd-9143-9a145dc59cf4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"7a83b820-6990-40dd-9143-9a145dc59cf4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"7a83b820-6990-40dd-9143-9a145dc59cf4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f31fe682-0338-46c3-90cc-8d497601b809","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"7a83b820-6990-40dd-9143-9a145dc59cf4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"7a83b820-6990-40dd-9143-9a145dc59cf4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"7a83b820-6990-40dd-9143-9a145dc59cf4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"eb7000ee-d49b-4e77-81fd-1489fa885858","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"eb7000ee-d49b-4e77-81fd-1489fa885858","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"eb7000ee-d49b-4e77-81fd-1489fa885858","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"reconcile","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"9dd8829d-3700-4d4e-b064-b04d985d305b"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"apply","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"9dd8829d-3700-4d4e-b064-b04d985d305b"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"9dd8829d-3700-4d4e-b064-b04d985d305b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.renderMaasOperatorInstall"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"20907ff2-e58a-4eb4-ad58-e5c3ba05849f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trustyai.initialize"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"20907ff2-e58a-4eb4-ad58-e5c3ba05849f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trustyai.createConfigMap"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"20907ff2-e58a-4eb4-ad58-e5c3ba05849f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"20907ff2-e58a-4eb4-ad58-e5c3ba05849f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"20907ff2-e58a-4eb4-ad58-e5c3ba05849f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:15Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c27ec42a-f9be-465b-b579-67a3d46569bd"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c27ec42a-f9be-465b-b579-67a3d46569bd"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c27ec42a-f9be-465b-b579-67a3d46569bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c27ec42a-f9be-465b-b579-67a3d46569bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"9dd8829d-3700-4d4e-b064-b04d985d305b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"28b3b3c7-946e-4bcc-89a4-4e74b3de176e"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"apply","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"28b3b3c7-946e-4bcc-89a4-4e74b3de176e"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"28b3b3c7-946e-4bcc-89a4-4e74b3de176e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/ogx.initialize"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"28b3b3c7-946e-4bcc-89a4-4e74b3de176e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/ogx.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"c27ec42a-f9be-465b-b579-67a3d46569bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"28b3b3c7-946e-4bcc-89a4-4e74b3de176e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"reconcile","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"5cba3422-7e63-4a3c-9feb-d8469855c863"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"apply","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"5cba3422-7e63-4a3c-9feb-d8469855c863"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"reconcile","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"0916224f-df75-46ad-a625-d289a2119836"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"apply","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"0916224f-df75-46ad-a625-d289a2119836"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"reconcile","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2295b2c1-8882-4e58-a555-fa1c4fa5d4fd"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"apply","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2295b2c1-8882-4e58-a555-fa1c4fa5d4fd"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"reconcile","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"81402cb8-f793-4078-b495-6992e381964d"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"apply","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"81402cb8-f793-4078-b495-6992e381964d"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4b491470-d8a3-43ee-8a87-48809df4f6b3"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4b491470-d8a3-43ee-8a87-48809df4f6b3"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"reconcile","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"5f6cb5a8-99d4-4d0a-9a9b-88a594842abe"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"apply","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"5f6cb5a8-99d4-4d0a-9a9b-88a594842abe"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"81402cb8-f793-4078-b495-6992e381964d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.setKustomizedParams"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"81402cb8-f793-4078-b495-6992e381964d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.configureDependencies"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"81402cb8-f793-4078-b495-6992e381964d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"81402cb8-f793-4078-b495-6992e381964d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"5f6cb5a8-99d4-4d0a-9a9b-88a594842abe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"5f6cb5a8-99d4-4d0a-9a9b-88a594842abe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4b491470-d8a3-43ee-8a87-48809df4f6b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4b491470-d8a3-43ee-8a87-48809df4f6b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4b491470-d8a3-43ee-8a87-48809df4f6b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4b491470-d8a3-43ee-8a87-48809df4f6b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T09:48:16Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4b491470-d8a3-43ee-8a87-48809df4f6b3","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4b491470-d8a3-43ee-8a87-48809df4f6b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:16Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4b491470-d8a3-43ee-8a87-48809df4f6b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4b491470-d8a3-43ee-8a87-48809df4f6b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2295b2c1-8882-4e58-a555-fa1c4fa5d4fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/ray.initialize"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2295b2c1-8882-4e58-a555-fa1c4fa5d4fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2295b2c1-8882-4e58-a555-fa1c4fa5d4fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2295b2c1-8882-4e58-a555-fa1c4fa5d4fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4b491470-d8a3-43ee-8a87-48809df4f6b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"4b491470-d8a3-43ee-8a87-48809df4f6b3","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2295b2c1-8882-4e58-a555-fa1c4fa5d4fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2295b2c1-8882-4e58-a555-fa1c4fa5d4fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:17Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"2295b2c1-8882-4e58-a555-fa1c4fa5d4fd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm","msg":"delete","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b5e3b98b-ddb0-406b-989a-24dd9bcf35fc","gvk":"components.platform.opendatahub.io/v1alpha1, Kind=SparkOperator","ns":"","name":"default-sparkoperator"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c7c942a5-b2b2-48e2-8c76-dcb5a681108a"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"fab52f3e-a771-4856-99e2-3d2575d9d33e"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"fbdcf354-4049-4627-a85c-26da831bf487"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c7437ba8-25c8-46f5-8331-d52a42824333"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"82de3e42-f9d2-4be8-ae38-ce69d6d299cb"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"2fc76f84-04ea-4c06-884e-6a268439133d"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"715b2af4-8a26-42bb-b243-b128905a78f8"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"73633869-3149-4054-8164-612e53b0a824"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0a2d1fbc-d814-436c-875d-672f96418899"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c88f888c-a7a7-4f9b-b6d2-0629a0af08dc"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"3e152316-514a-4d76-8f08-6cbf74d07315","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"8d87b9df-9f10-41ae-8efb-e3c996844724"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0780174d-e28b-4eef-b8bd-4c707a8682b6"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"2b376591-effa-48aa-aa5a-24135b48d8f1"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b5e3b98b-ddb0-406b-989a-24dd9bcf35fc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"412bc73d-d36b-435f-9a3a-5944038ef295"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"c783e495-abd4-47a0-adc8-fc74f0d8b215"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b1c67b5e-0ed6-45f2-9757-c10ecd62234f"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b1c67b5e-0ed6-45f2-9757-c10ecd62234f"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b1c67b5e-0ed6-45f2-9757-c10ecd62234f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b1c67b5e-0ed6-45f2-9757-c10ecd62234f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b1c67b5e-0ed6-45f2-9757-c10ecd62234f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:48:18Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b1c67b5e-0ed6-45f2-9757-c10ecd62234f","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b1c67b5e-0ed6-45f2-9757-c10ecd62234f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b1c67b5e-0ed6-45f2-9757-c10ecd62234f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"190ac41f-a55b-4cbd-9460-e77941290115"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"9015c0e2-2d30-40f5-a2dc-ebae0a59c441"} {"level":"info","ts":"2026-07-06T09:48:18Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"4938bcec-5c0e-4018-b61c-f19757c94ec8"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"0ab04801-a8ab-4b86-9b4e-cbbe4b022c16"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"1fb422a6-48ff-4af9-9133-f50da0dbad36"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"7a019844-21a9-4cb8-9204-d3731635a7d1"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.reconcileModelCache"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"a1684a75-14d9-44b3-b10d-5774bbeeb784","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:48:19Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"38dc1e90-4aaa-49d0-9240-12eab1a28b08","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"38dc1e90-4aaa-49d0-9240-12eab1a28b08","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"5b7fb205-8688-487d-949b-f50543738e66","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b1c67b5e-0ed6-45f2-9757-c10ecd62234f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"4a2eb736-6c8b-4305-84e7-f0a8aa9ad9df","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"4a2eb736-6c8b-4305-84e7-f0a8aa9ad9df","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"0916224f-df75-46ad-a625-d289a2119836","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"b1c67b5e-0ed6-45f2-9757-c10ecd62234f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"0916224f-df75-46ad-a625-d289a2119836","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"0916224f-df75-46ad-a625-d289a2119836","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/imagestreams.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"0916224f-df75-46ad-a625-d289a2119836","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.updateStatus"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"0916224f-df75-46ad-a625-d289a2119836","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ba791791-3e09-4c5f-8cd0-b9dc435e0cbd"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ba791791-3e09-4c5f-8cd0-b9dc435e0cbd"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ba791791-3e09-4c5f-8cd0-b9dc435e0cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ba791791-3e09-4c5f-8cd0-b9dc435e0cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ba791791-3e09-4c5f-8cd0-b9dc435e0cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:48:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ba791791-3e09-4c5f-8cd0-b9dc435e0cbd","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ba791791-3e09-4c5f-8cd0-b9dc435e0cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ba791791-3e09-4c5f-8cd0-b9dc435e0cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"de0f7d2a-2bfa-4c1a-ac49-4a969dffc47b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ba791791-3e09-4c5f-8cd0-b9dc435e0cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-06T09:48:19Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"ba791791-3e09-4c5f-8cd0-b9dc435e0cbd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:19Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f99fc0ba-b6b8-4718-828e-aead2cea3b35","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"81402cb8-f793-4078-b495-6992e381964d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"81402cb8-f793-4078-b495-6992e381964d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.reconcileHardwareProfiles"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"81402cb8-f793-4078-b495-6992e381964d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.updateStatus"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"81402cb8-f793-4078-b495-6992e381964d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"81402cb8-f793-4078-b495-6992e381964d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"4a8d5e26-bbef-4902-826b-a88d724d20c8","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T09:48:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a521a7b3-b59a-4168-ad68-656f98c78f0a"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a521a7b3-b59a-4168-ad68-656f98c78f0a"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a521a7b3-b59a-4168-ad68-656f98c78f0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a521a7b3-b59a-4168-ad68-656f98c78f0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a521a7b3-b59a-4168-ad68-656f98c78f0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:48:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a521a7b3-b59a-4168-ad68-656f98c78f0a","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a521a7b3-b59a-4168-ad68-656f98c78f0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a521a7b3-b59a-4168-ad68-656f98c78f0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"564f3e2c-9859-4511-97fe-b3041eae92e5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a521a7b3-b59a-4168-ad68-656f98c78f0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"0916224f-df75-46ad-a625-d289a2119836","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:20Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a521a7b3-b59a-4168-ad68-656f98c78f0a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-06T09:48:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"d27f4dd0-8836-4c93-bb5d-b025f2bf3c57","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13148471-8937-4c1d-9371-c04123cd31d5"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13148471-8937-4c1d-9371-c04123cd31d5"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13148471-8937-4c1d-9371-c04123cd31d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13148471-8937-4c1d-9371-c04123cd31d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13148471-8937-4c1d-9371-c04123cd31d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:48:21Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13148471-8937-4c1d-9371-c04123cd31d5","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13148471-8937-4c1d-9371-c04123cd31d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13148471-8937-4c1d-9371-c04123cd31d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13148471-8937-4c1d-9371-c04123cd31d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:48:21Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13148471-8937-4c1d-9371-c04123cd31d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-06T09:48:41Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T09:49:20Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a094d03f-620b-4fd0-81d8-a31a1efc2fdc"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a094d03f-620b-4fd0-81d8-a31a1efc2fdc"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a094d03f-620b-4fd0-81d8-a31a1efc2fdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a094d03f-620b-4fd0-81d8-a31a1efc2fdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a094d03f-620b-4fd0-81d8-a31a1efc2fdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.initializeModules"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.cleanupDisabledModules"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.provisionModules"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/helm.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a094d03f-620b-4fd0-81d8-a31a1efc2fdc","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.injectModuleEnv"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/modules.updateModuleStatus"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a094d03f-620b-4fd0-81d8-a31a1efc2fdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a094d03f-620b-4fd0-81d8-a31a1efc2fdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"97862714-21cd-467d-88a1-2e94a8b64961","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modules","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"773afd2c-d35c-4b9b-9564-7602f933771f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/dynamicownership.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"c8796587-db42-4444-b7bd-684e5f35ac74"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"c8796587-db42-4444-b7bd-684e5f35ac74"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"c8796587-db42-4444-b7bd-684e5f35ac74","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trustyai.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a0f743b5-3897-4594-b813-cc1787e7426f"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a0f743b5-3897-4594-b813-cc1787e7426f"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Pre-condition skipped by runtime predicate","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a0f743b5-3897-4594-b813-cc1787e7426f","conditionType":"LLM-D-WVADependencies"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a0f743b5-3897-4594-b813-cc1787e7426f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelcontroller.initialize"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a0f743b5-3897-4594-b813-cc1787e7426f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a0f743b5-3897-4594-b813-cc1787e7426f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"5d9753eb-0006-47c1-96fa-0c0577b832d5"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"5d9753eb-0006-47c1-96fa-0c0577b832d5"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"5d9753eb-0006-47c1-96fa-0c0577b832d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.renderMaasOperatorInstall"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"5d9753eb-0006-47c1-96fa-0c0577b832d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"9a18077e-f376-4622-9764-4e2afe7a4c20"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"9a18077e-f376-4622-9764-4e2afe7a4c20"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"9a18077e-f376-4622-9764-4e2afe7a4c20","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.initialize"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"9a18077e-f376-4622-9764-4e2afe7a4c20","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainer.checkJobSetCRD"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"c8796587-db42-4444-b7bd-684e5f35ac74","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trustyai.initialize"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"c8796587-db42-4444-b7bd-684e5f35ac74","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trustyai.createConfigMap"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"c8796587-db42-4444-b7bd-684e5f35ac74","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"c8796587-db42-4444-b7bd-684e5f35ac74","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"c8796587-db42-4444-b7bd-684e5f35ac74","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"72f2a86a-f971-48ae-80fb-2e78e8268e99"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"72f2a86a-f971-48ae-80fb-2e78e8268e99"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"72f2a86a-f971-48ae-80fb-2e78e8268e99","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/ogx.initialize"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"72f2a86a-f971-48ae-80fb-2e78e8268e99","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/ogx.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"72f2a86a-f971-48ae-80fb-2e78e8268e99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"72f2a86a-f971-48ae-80fb-2e78e8268e99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"72f2a86a-f971-48ae-80fb-2e78e8268e99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"72f2a86a-f971-48ae-80fb-2e78e8268e99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"72f2a86a-f971-48ae-80fb-2e78e8268e99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"ogx","controllerGroup":"components.platform.opendatahub.io","controllerKind":"OGX","OGX":{"name":"default-ogx"},"namespace":"","name":"default-ogx","reconcileID":"72f2a86a-f971-48ae-80fb-2e78e8268e99","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"8e38aab7-993a-4f60-8bf0-adff87ebce7d"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"8e38aab7-993a-4f60-8bf0-adff87ebce7d"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"8e38aab7-993a-4f60-8bf0-adff87ebce7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/mlflowoperator.initialize"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"8e38aab7-993a-4f60-8bf0-adff87ebce7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/mlflowoperator.setKustomizedParams"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"8e38aab7-993a-4f60-8bf0-adff87ebce7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"8e38aab7-993a-4f60-8bf0-adff87ebce7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"9a18077e-f376-4622-9764-4e2afe7a4c20","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"9a18077e-f376-4622-9764-4e2afe7a4c20","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"8e38aab7-993a-4f60-8bf0-adff87ebce7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"9a18077e-f376-4622-9764-4e2afe7a4c20","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.initialize"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.configureDependencies"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.setKustomizedParams"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"8e38aab7-993a-4f60-8bf0-adff87ebce7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"8e38aab7-993a-4f60-8bf0-adff87ebce7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"mlflowoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"MLflowOperator","MLflowOperator":{"name":"default-mlflowoperator"},"namespace":"","name":"default-mlflowoperator","reconcileID":"8e38aab7-993a-4f60-8bf0-adff87ebce7d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"1ac62f7a-b62e-4225-9856-4cae68bb7cfa"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"1ac62f7a-b62e-4225-9856-4cae68bb7cfa"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"1ac62f7a-b62e-4225-9856-4cae68bb7cfa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/sanitycheck.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a0f743b5-3897-4594-b813-cc1787e7426f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a0f743b5-3897-4594-b813-cc1787e7426f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modelcontroller","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelController","ModelController":{"name":"default-modelcontroller"},"namespace":"","name":"default-modelcontroller","reconcileID":"a0f743b5-3897-4594-b813-cc1787e7426f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"9a18077e-f376-4622-9764-4e2afe7a4c20","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"9a18077e-f376-4622-9764-4e2afe7a4c20","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trainer","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Trainer","Trainer":{"name":"default-trainer"},"namespace":"","name":"default-trainer","reconcileID":"9a18077e-f376-4622-9764-4e2afe7a4c20","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"69ec6457-9a07-4fbe-993f-58013292e861"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"69ec6457-9a07-4fbe-993f-58013292e861"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"69ec6457-9a07-4fbe-993f-58013292e861","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"69ec6457-9a07-4fbe-993f-58013292e861","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"69ec6457-9a07-4fbe-993f-58013292e861","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"1ac62f7a-b62e-4225-9856-4cae68bb7cfa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/ray.initialize"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"1ac62f7a-b62e-4225-9856-4cae68bb7cfa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"1ac62f7a-b62e-4225-9856-4cae68bb7cfa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"5d9753eb-0006-47c1-96fa-0c0577b832d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.ensureMaasClusterConfigControllerRef"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"5d9753eb-0006-47c1-96fa-0c0577b832d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"5d9753eb-0006-47c1-96fa-0c0577b832d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"5d9753eb-0006-47c1-96fa-0c0577b832d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"c8796587-db42-4444-b7bd-684e5f35ac74","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"c8796587-db42-4444-b7bd-684e5f35ac74","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"trustyai","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrustyAI","TrustyAI":{"name":"default-trustyai"},"namespace":"","name":"default-trustyai","reconcileID":"c8796587-db42-4444-b7bd-684e5f35ac74","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:49:36Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"10d6eb8b-4a29-4f83-a62e-2d78d7a7caa0","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"10d6eb8b-4a29-4f83-a62e-2d78d7a7caa0","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.initialize"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.deployObservabilityManifests"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920534e7-d3f9-4317-95df-c21fcd2b75bd"} {"level":"info","ts":"2026-07-06T09:49:36Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920534e7-d3f9-4317-95df-c21fcd2b75bd"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"reconcile","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"65cf2e5d-900b-4b2d-8a0e-4276756db7f9"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"apply","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"65cf2e5d-900b-4b2d-8a0e-4276756db7f9"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"reconcile","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"apply","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"reconcile","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"apply","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.initialize"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.customizeManifests"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.configureDependencies"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0c573a1-6520-4ded-be03-b395cf4503ce","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"reconcile","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"888ec2df-07cc-4720-98d1-8b4f90243a5e"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"apply","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"888ec2df-07cc-4720-98d1-8b4f90243a5e"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"888ec2df-07cc-4720-98d1-8b4f90243a5e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/trainingoperator.initialize"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"888ec2df-07cc-4720-98d1-8b4f90243a5e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"888ec2df-07cc-4720-98d1-8b4f90243a5e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"888ec2df-07cc-4720-98d1-8b4f90243a5e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"reconcile","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"d4169fec-c814-4da9-942f-a0a6089b182e"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"apply","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"d4169fec-c814-4da9-942f-a0a6089b182e"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"d4169fec-c814-4da9-942f-a0a6089b182e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/feastoperator.initialize"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"d4169fec-c814-4da9-942f-a0a6089b182e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/feastoperator.setKustomizedParams"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"d4169fec-c814-4da9-942f-a0a6089b182e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"d4169fec-c814-4da9-942f-a0a6089b182e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:37Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"d4169fec-c814-4da9-942f-a0a6089b182e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/feastoperator.migrateDeploymentSelector"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"1ac62f7a-b62e-4225-9856-4cae68bb7cfa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"1ac62f7a-b62e-4225-9856-4cae68bb7cfa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"ray","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Ray","Ray":{"name":"default-ray"},"namespace":"","name":"default-ray","reconcileID":"1ac62f7a-b62e-4225-9856-4cae68bb7cfa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920534e7-d3f9-4317-95df-c21fcd2b75bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920534e7-d3f9-4317-95df-c21fcd2b75bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920534e7-d3f9-4317-95df-c21fcd2b75bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920534e7-d3f9-4317-95df-c21fcd2b75bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T09:49:38Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920534e7-d3f9-4317-95df-c21fcd2b75bd","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920534e7-d3f9-4317-95df-c21fcd2b75bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920534e7-d3f9-4317-95df-c21fcd2b75bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a094d03f-620b-4fd0-81d8-a31a1efc2fdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.setKustomizedParams"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.configureDependencies"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920534e7-d3f9-4317-95df-c21fcd2b75bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"65cf2e5d-900b-4b2d-8a0e-4276756db7f9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"65cf2e5d-900b-4b2d-8a0e-4276756db7f9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"datasciencepipelines","controllerGroup":"components.platform.opendatahub.io","controllerKind":"DataSciencePipelines","DataSciencePipelines":{"name":"default-datasciencepipelines"},"namespace":"","name":"default-datasciencepipelines","reconcileID":"65cf2e5d-900b-4b2d-8a0e-4276756db7f9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"d4169fec-c814-4da9-942f-a0a6089b182e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920534e7-d3f9-4317-95df-c21fcd2b75bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"920534e7-d3f9-4317-95df-c21fcd2b75bd","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.initialize"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.removeOwnershipFromUnmanagedResources"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.cleanUpTemplatedResources"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"888ec2df-07cc-4720-98d1-8b4f90243a5e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"888ec2df-07cc-4720-98d1-8b4f90243a5e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"trainingoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"TrainingOperator","TrainingOperator":{"name":"default-trainingoperator"},"namespace":"","name":"default-trainingoperator","reconcileID":"888ec2df-07cc-4720-98d1-8b4f90243a5e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelregistry.updateStatus"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:38Z","msg":"Executing action","controller":"modelregistry","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelRegistry","ModelRegistry":{"name":"default-modelregistry"},"namespace":"","name":"default-modelregistry","reconcileID":"e99cacc3-86f6-4efb-99a5-547a6aa14a4f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:39Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.customizeKserveConfigMap"} {"level":"info","ts":"2026-07-06T09:49:39Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-06T09:49:39Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:39Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"d4169fec-c814-4da9-942f-a0a6089b182e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:39Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"69ec6457-9a07-4fbe-993f-58013292e861","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:39Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"d4169fec-c814-4da9-942f-a0a6089b182e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:39Z","msg":"Executing action","controller":"feastoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"FeastOperator","FeastOperator":{"name":"default-feastoperator"},"namespace":"","name":"default-feastoperator","reconcileID":"d4169fec-c814-4da9-942f-a0a6089b182e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a094d03f-620b-4fd0-81d8-a31a1efc2fdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb6a1f2f-9a20-4106-9650-51179481c844"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb6a1f2f-9a20-4106-9650-51179481c844"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb6a1f2f-9a20-4106-9650-51179481c844","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb6a1f2f-9a20-4106-9650-51179481c844","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb6a1f2f-9a20-4106-9650-51179481c844","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:49:40Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb6a1f2f-9a20-4106-9650-51179481c844","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb6a1f2f-9a20-4106-9650-51179481c844","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb6a1f2f-9a20-4106-9650-51179481c844","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"0b618ead-5d85-4f97-b393-b3285a3f7be1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:40Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kserve.reconcileModelCache"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"kserve","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kserve","Kserve":{"name":"default-kserve"},"namespace":"","name":"default-kserve","reconcileID":"d586e040-265a-4908-a48c-cc114485ce9c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/imagestreams.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/workbenches.updateStatus"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c89b64ac-e200-4883-b2ef-eb77b2a840fa","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb6a1f2f-9a20-4106-9650-51179481c844","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c89b64ac-e200-4883-b2ef-eb77b2a840fa","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"eb6a1f2f-9a20-4106-9650-51179481c844","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"cd7aca86-8d7b-4b32-80bb-2d5547d90121","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"69ec6457-9a07-4fbe-993f-58013292e861","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"69ec6457-9a07-4fbe-993f-58013292e861","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"99ffecf1-18d9-43df-ad99-e5c4db7ea8dd","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"99ffecf1-18d9-43df-ad99-e5c4db7ea8dd","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-06T09:49:41Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"workbenches","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Workbenches","Workbenches":{"name":"default-workbenches"},"namespace":"","name":"default-workbenches","reconcileID":"f0c13dfd-d716-4205-8b2c-39dac104f108","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.reconcileHardwareProfiles"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/dashboard.updateStatus"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"dashboard","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Dashboard","Dashboard":{"name":"default-dashboard"},"namespace":"","name":"default-dashboard","reconcileID":"4a54675d-ab9e-44ea-a650-9c49ec51f3ec","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T09:49:41Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"30ccf4e2-d464-4a93-9a13-ef7352c7b8fb","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"69ec6457-9a07-4fbe-993f-58013292e861","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"53b2596f-0fd3-44b9-b8e0-ddf19f0b03d5"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"53b2596f-0fd3-44b9-b8e0-ddf19f0b03d5"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"53b2596f-0fd3-44b9-b8e0-ddf19f0b03d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"53b2596f-0fd3-44b9-b8e0-ddf19f0b03d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"53b2596f-0fd3-44b9-b8e0-ddf19f0b03d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"7a2c621a-db56-4a6d-af29-a506cf8fd35f"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"7a2c621a-db56-4a6d-af29-a506cf8fd35f"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"7a2c621a-db56-4a6d-af29-a506cf8fd35f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"7a2c621a-db56-4a6d-af29-a506cf8fd35f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"7a2c621a-db56-4a6d-af29-a506cf8fd35f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"53b2596f-0fd3-44b9-b8e0-ddf19f0b03d5","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"53b2596f-0fd3-44b9-b8e0-ddf19f0b03d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"53b2596f-0fd3-44b9-b8e0-ddf19f0b03d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"7a2c621a-db56-4a6d-af29-a506cf8fd35f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:41Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"ea8d53c2-c1b6-4cf7-8b25-8cd50a6f15ab","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"53b2596f-0fd3-44b9-b8e0-ddf19f0b03d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"7a2c621a-db56-4a6d-af29-a506cf8fd35f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"7a2c621a-db56-4a6d-af29-a506cf8fd35f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"7a2c621a-db56-4a6d-af29-a506cf8fd35f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"53b2596f-0fd3-44b9-b8e0-ddf19f0b03d5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e1cc0c03-ae26-4a3c-b026-dd643f89f8bf"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e1cc0c03-ae26-4a3c-b026-dd643f89f8bf"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e1cc0c03-ae26-4a3c-b026-dd643f89f8bf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e1cc0c03-ae26-4a3c-b026-dd643f89f8bf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e1cc0c03-ae26-4a3c-b026-dd643f89f8bf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:49:42Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e1cc0c03-ae26-4a3c-b026-dd643f89f8bf","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e1cc0c03-ae26-4a3c-b026-dd643f89f8bf","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e1cc0c03-ae26-4a3c-b026-dd643f89f8bf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e1cc0c03-ae26-4a3c-b026-dd643f89f8bf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:42Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"e1cc0c03-ae26-4a3c-b026-dd643f89f8bf","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"fc152068-9c44-433d-8877-b20fd15755ba"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"fc152068-9c44-433d-8877-b20fd15755ba"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"fc152068-9c44-433d-8877-b20fd15755ba","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"fc152068-9c44-433d-8877-b20fd15755ba","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"fc152068-9c44-433d-8877-b20fd15755ba","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"fc152068-9c44-433d-8877-b20fd15755ba","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"fc152068-9c44-433d-8877-b20fd15755ba","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"fc152068-9c44-433d-8877-b20fd15755ba","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"fc152068-9c44-433d-8877-b20fd15755ba","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"667f1109-359b-447a-aec3-171bbc205c3b"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"667f1109-359b-447a-aec3-171bbc205c3b"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"667f1109-359b-447a-aec3-171bbc205c3b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"667f1109-359b-447a-aec3-171bbc205c3b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"667f1109-359b-447a-aec3-171bbc205c3b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:49:53Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"667f1109-359b-447a-aec3-171bbc205c3b","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"667f1109-359b-447a-aec3-171bbc205c3b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"667f1109-359b-447a-aec3-171bbc205c3b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"667f1109-359b-447a-aec3-171bbc205c3b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:53Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"667f1109-359b-447a-aec3-171bbc205c3b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"9ce2578f-d2b2-44bd-a6dd-b70a6b5e8496"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"9ce2578f-d2b2-44bd-a6dd-b70a6b5e8496"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"9ce2578f-d2b2-44bd-a6dd-b70a6b5e8496","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"9ce2578f-d2b2-44bd-a6dd-b70a6b5e8496","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"9ce2578f-d2b2-44bd-a6dd-b70a6b5e8496","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"9ce2578f-d2b2-44bd-a6dd-b70a6b5e8496","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"9ce2578f-d2b2-44bd-a6dd-b70a6b5e8496","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"9ce2578f-d2b2-44bd-a6dd-b70a6b5e8496","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"9ce2578f-d2b2-44bd-a6dd-b70a6b5e8496","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92fd7a8a-9b6c-46bd-af30-9c8ebc3f1861"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92fd7a8a-9b6c-46bd-af30-9c8ebc3f1861"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92fd7a8a-9b6c-46bd-af30-9c8ebc3f1861","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92fd7a8a-9b6c-46bd-af30-9c8ebc3f1861","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92fd7a8a-9b6c-46bd-af30-9c8ebc3f1861","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:49:54Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92fd7a8a-9b6c-46bd-af30-9c8ebc3f1861","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92fd7a8a-9b6c-46bd-af30-9c8ebc3f1861","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:49:54Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92fd7a8a-9b6c-46bd-af30-9c8ebc3f1861","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92fd7a8a-9b6c-46bd-af30-9c8ebc3f1861","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"92fd7a8a-9b6c-46bd-af30-9c8ebc3f1861","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"6f2743bf-b5bd-4772-b01e-b4a8b9fc7e28","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:55Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"f2eb79c8-ea97-4b12-986d-2f2ee5e89fd2","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T09:49:55Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"f2eb79c8-ea97-4b12-986d-2f2ee5e89fd2","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e19cd21a-108e-4066-af4d-7f09a6bb4e9f"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e19cd21a-108e-4066-af4d-7f09a6bb4e9f"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e19cd21a-108e-4066-af4d-7f09a6bb4e9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e19cd21a-108e-4066-af4d-7f09a6bb4e9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e19cd21a-108e-4066-af4d-7f09a6bb4e9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e19cd21a-108e-4066-af4d-7f09a6bb4e9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e19cd21a-108e-4066-af4d-7f09a6bb4e9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e19cd21a-108e-4066-af4d-7f09a6bb4e9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"e19cd21a-108e-4066-af4d-7f09a6bb4e9f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a947fe25-bdd5-4299-b50b-8d84b76e355e"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a947fe25-bdd5-4299-b50b-8d84b76e355e"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a947fe25-bdd5-4299-b50b-8d84b76e355e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a947fe25-bdd5-4299-b50b-8d84b76e355e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a947fe25-bdd5-4299-b50b-8d84b76e355e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:49:58Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a947fe25-bdd5-4299-b50b-8d84b76e355e","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a947fe25-bdd5-4299-b50b-8d84b76e355e","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a947fe25-bdd5-4299-b50b-8d84b76e355e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"reconcile","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"a6f2856c-e88c-4b50-9fb6-30da472f39f1"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"apply","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"a6f2856c-e88c-4b50-9fb6-30da472f39f1"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"a6f2856c-e88c-4b50-9fb6-30da472f39f1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/sparkoperator.initialize"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"a6f2856c-e88c-4b50-9fb6-30da472f39f1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"a6f2856c-e88c-4b50-9fb6-30da472f39f1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"a6f2856c-e88c-4b50-9fb6-30da472f39f1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a947fe25-bdd5-4299-b50b-8d84b76e355e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"a6f2856c-e88c-4b50-9fb6-30da472f39f1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"a6f2856c-e88c-4b50-9fb6-30da472f39f1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"sparkoperator","controllerGroup":"components.platform.opendatahub.io","controllerKind":"SparkOperator","SparkOperator":{"name":"default-sparkoperator"},"namespace":"","name":"default-sparkoperator","reconcileID":"a6f2856c-e88c-4b50-9fb6-30da472f39f1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:49:58Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"a947fe25-bdd5-4299-b50b-8d84b76e355e","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-06T09:50:09Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T09:50:20Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:50:20Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"bfcc2238-daa8-422e-b9a0-f36912e0ddd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T09:50:35Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:50:35Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"40ba8097-ca5f-4ec1-b337-472372c26400","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:50:46Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-06T09:50:56Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-06T09:50:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-06T09:51:12Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-lqz59","reconcileID":"d8270244-3554-453d-b4e7-d7c6009f9c38"} {"level":"info","ts":"2026-07-06T09:51:12Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-lqz59","reconcileID":"a2d609a4-3b73-405a-a0d3-2c04df6fbc05"} {"level":"info","ts":"2026-07-06T09:51:13Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-lqz59","reconcileID":"2571a778-4e73-4e87-8e4d-87639c58c6c5"} {"level":"info","ts":"2026-07-06T09:51:13Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-c0gy3","reconcileID":"6b3ba2ee-35ea-4013-b7c6-b1d476fa9ddb"} {"level":"info","ts":"2026-07-06T09:51:13Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-c0gy3","reconcileID":"1f7c8330-878d-439f-9237-cffefa188e9b"} {"level":"info","ts":"2026-07-06T09:51:13Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-c0gy3","reconcileID":"dd8e6cb0-c2f0-4979-bbaa-4c02fc7f2d25"} {"level":"info","ts":"2026-07-06T09:51:15Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1"} {"level":"info","ts":"2026-07-06T09:51:15Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1"} {"level":"info","ts":"2026-07-06T09:51:15Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:51:15Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-06T09:51:15Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:15Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:15Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-06T09:51:16Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"2bd357d7-22a2-4292-a3f7-b8ab3d0754a1","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d28f21c7-35e4-424d-873e-2ebc5eadab19"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d28f21c7-35e4-424d-873e-2ebc5eadab19"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d28f21c7-35e4-424d-873e-2ebc5eadab19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d28f21c7-35e4-424d-873e-2ebc5eadab19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d28f21c7-35e4-424d-873e-2ebc5eadab19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:51:16Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d28f21c7-35e4-424d-873e-2ebc5eadab19","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d28f21c7-35e4-424d-873e-2ebc5eadab19","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d28f21c7-35e4-424d-873e-2ebc5eadab19","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d28f21c7-35e4-424d-873e-2ebc5eadab19","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"d28f21c7-35e4-424d-873e-2ebc5eadab19","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-06T09:51:16Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-06T09:51:16Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"298fbd6e-09a0-4437-a143-c6663ff8a2d9","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"33744e24-67b4-4de4-84ea-ba7d32ef224f"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"33744e24-67b4-4de4-84ea-ba7d32ef224f"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"33744e24-67b4-4de4-84ea-ba7d32ef224f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"33744e24-67b4-4de4-84ea-ba7d32ef224f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"33744e24-67b4-4de4-84ea-ba7d32ef224f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-06T09:51:17Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus.checkMaaSPrerequisites","msg":"MaaS gateway missing required annotations","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"33744e24-67b4-4de4-84ea-ba7d32ef224f","gateway":"maas-default-gateway","missing":["security.opendatahub.io/authorino-tls-bootstrap=\"true\""]} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"33744e24-67b4-4de4-84ea-ba7d32ef224f","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"33744e24-67b4-4de4-84ea-ba7d32ef224f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"33744e24-67b4-4de4-84ea-ba7d32ef224f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T09:51:17Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"33744e24-67b4-4de4-84ea-ba7d32ef224f","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T09:51:26Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-06T09:51:35Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} ==================== OPERATOR POD LOG: rhods-operator-6f6bb89fdf-f4zpw.log ==================== ==================== OPERATOR POD LOG: rhods-operator-6f6bb89fdf-kr2pd.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-6bd46884d4-lwx6j.log ==================== 2026/07/06 08:11:09 [notice] 1#1: using the "epoll" event method 2026/07/06 08:11:09 [notice] 1#1: nginx/1.26.3 2026/07/06 08:11:09 [notice] 1#1: built by gcc 11.5.0 20240719 (Red Hat 11.5.0-14) (GCC) 2026/07/06 08:11:09 [notice] 1#1: OS: Linux 5.14.0-570.60.1.el9_6.x86_64 2026/07/06 08:11:09 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2026/07/06 08:11:09 [notice] 1#1: start worker processes 2026/07/06 08:11:09 [notice] 1#1: start worker process 12 2026/07/06 08:11:09 [notice] 1#1: start worker process 13 ==================== WORKLOAD POD LOG: redhat-ods-applications/data-science-pipelines-operator-controller-manager-7fc7f768f5wq.log ==================== 2026-07-06T08:11:14Z INFO setup starting manager 2026-07-06T08:11:14Z INFO controller-runtime.metrics Starting metrics server 2026-07-06T08:11:14Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false} 2026-07-06T08:11:14Z INFO starting server {"name": "health probe", "addr": "[::]:8081"} I0706 08:11:14.662065 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="redhat-ods-applications/f9eb95d5.opendatahub.io" I0706 08:12:23.263826 1 leaderelection.go:272] "Successfully acquired lease" lock="redhat-ods-applications/f9eb95d5.opendatahub.io" 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Secret"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Deployment"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Secret"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.ConfigMap"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.DataSciencePipelinesApplication"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Service"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.ServiceAccount"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.PersistentVolumeClaim"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.ConfigMap"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Pod"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.NetworkPolicy"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Role"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.RoleBinding"} 2026-07-06T08:12:23Z INFO Starting EventSource {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "source": "kind source: *v1.Route"} 2026-07-06T08:12:23Z INFO Starting Controller {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication"} 2026-07-06T08:12:23Z INFO Starting workers {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "worker count": 10} 2026-07-06T08:30:43Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Unable to connect to Database: dial tcp 172.30.214.66:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:30:44Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:44Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:44Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Unable to connect to Database: dial tcp 172.30.214.66:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:30:44Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:44Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Unable to connect to Database: dial tcp 172.30.214.66:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:44Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:44Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:44Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:30:44Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:44Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:45Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Unable to connect to Database: dial tcp 172.30.214.66:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:45Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:30:45Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:45Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:45Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:45Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:45Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:45Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:45Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Unable to connect to Database: dial tcp 172.30.214.66:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:45Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:45Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:30:45Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:45Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:45Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:45Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:45Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:45Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:46Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Unable to connect to Database: dial tcp 172.30.214.66:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:46Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:46Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:30:46Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:46Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:46Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:46Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:46Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:46Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Unable to connect to Database: dial tcp 172.30.214.66:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:46Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:46Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:46Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:30:46Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:46Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:46Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:46Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:46Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Unable to connect to Database: dial tcp 172.30.214.66:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:30:55Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:55Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:55Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Unable to connect to Database: dial tcp 172.30.214.66:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:30:55Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:55Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:55Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Unable to connect to Database: dial tcp 172.30.214.66:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:55Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:55Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:30:55Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Unable to connect to Database: dial tcp 172.30.214.66:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:30:55Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:30:55Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:30:55Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:30:55Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:06Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:06Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:06Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:06Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:06Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:06Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:06Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:06Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:06Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:06Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:07Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:07Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:07Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:07Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:07Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:07Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:07Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:08Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:08Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:08Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:08Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:08Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:08Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:08Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:08Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:08Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:09Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:09Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:09Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:09Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:09Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:09Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:09Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:15Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:16Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:16Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:16Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:16Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:16Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:16Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:16Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:16Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:16Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:16Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:16Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:16Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:16Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:16Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:16Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:16Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:21Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:21Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:21Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:21Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:21Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:21Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:21Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:21Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:21Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:21Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:21Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:21Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:21Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:21Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:21Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:21Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:21Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:26Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:26Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:26Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:26Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:26Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:26Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:26Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:26Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:26Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:27Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:27Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:27Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:27Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:27Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:27Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:27Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:27Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:27Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:27Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:27Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:27Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:27Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:27Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:27Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:27Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:27Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:29Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:29Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:29Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:29Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:29Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:29Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:29Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:29Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:31:29Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:31:35Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:31:35Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:11Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:11Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:11Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:11Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:11Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:11Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:11Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:11Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:11Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:11Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:11Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:11Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:11Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:11Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:13Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:13Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:13Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:13Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:13Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:13Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:13Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:13Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:13Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:13Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:13Z INFO No other DSPAs with PipelineStore 'kubernetes' found. Cleaning up webhook resources. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T08:32:14Z INFO Webhook resources cleanup complete. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T08:32:14Z INFO DSPA resources cleaned up. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:14Z INFO No other DSPAs with PipelineStore 'kubernetes' found. Cleaning up webhook resources. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T08:32:14Z INFO Webhook resources cleanup complete. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T08:32:14Z INFO DataSciencePipelinesApplication.datasciencepipelinesapplications.opendatahub.io "dspa" not found {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:14Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "d4f30230-de35-4fe5-9dd2-eda895699436", "error": "Operation cannot be fulfilled on datasciencepipelinesapplications.datasciencepipelinesapplications.opendatahub.io \"dspa\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/datasciencepipelinesapplications.opendatahub.io/datasciencepipelinesapplications/cy-e2e-automl-binary-522120/dspa, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 7c77511e-50e7-4a85-8880-b89ab82c7660, UID in object meta: "} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:32:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Unable to connect to Database: dial tcp 172.30.66.102:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:52Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:52Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:32:52Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Unable to connect to Database: dial tcp 172.30.66.102:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:52Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:52Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:32:52Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Unable to connect to Database: dial tcp 172.30.66.102:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:32:52Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:52Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Unable to connect to Database: dial tcp 172.30.66.102:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:52Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:32:52Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:52Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:52Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:53Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:53Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:53Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:53Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:53Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:53Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:53Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:53Z INFO Unable to connect to Database: dial tcp 172.30.66.102:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:53Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:53Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:53Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:53Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:53Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:53Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:53Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:53Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:32:53Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Unable to connect to Database: dial tcp 172.30.66.102:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:54Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:54Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:54Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:32:54Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:54Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:54Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:54Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Unable to connect to Database: dial tcp 172.30.66.102:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:54Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:54Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:54Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:54Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:54Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:54Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:32:54Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:54Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:58Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Unable to connect to Database: dial tcp 172.30.66.102:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:58Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:58Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:58Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:58Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:32:58Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:58Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:58Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:58Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Unable to connect to Database: dial tcp 172.30.66.102:3306: connect: connection refused {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:32:58Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:58Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:58Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:58Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:32:58Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:32:58Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:58Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:32:58Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:33:09Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:09Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:09Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:09Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:09Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:09Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:09Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:09Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:09Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:09Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:09Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:09Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:09Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:10Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:10Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:10Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:10Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:10Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:10Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:10Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:10Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:10Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:10Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:10Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:10Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:10Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:10Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:10Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:10Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:10Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:10Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:10Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:11Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:11Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:11Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:11Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:11Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:11Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:11Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:11Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:11Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:11Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:11Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:11Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:12Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:12Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:12Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:12Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:12Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:12Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:12Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:12Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:13Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:13Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:13Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:13Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:13Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:13Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:13Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:13Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:13Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:16Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:16Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:16Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:16Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:16Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:16Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:16Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:16Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:16Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:17Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:17Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:17Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:17Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:17Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:17Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:17Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:17Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:17Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:17Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:17Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:17Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:18Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:18Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:18Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:18Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:18Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:18Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:18Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:18Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:18Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:18Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:18Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:18Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:19Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:19Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:19Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:19Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:19Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:19Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:33:19Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:19Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:25Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:25Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:25Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:25Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:25Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:25Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:25Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:25Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying Database Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying storage Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Performing Database Health Check {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Database Health Check Successful {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying Common Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Generating Sample Config {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying APIServer Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying PersistenceAgent Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying WorkflowController Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Finished applying MLMD Resources {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:33:26Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:33:26Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "True"} 2026-07-06T08:34:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace cy-e2e-automl-binary-522120, this bundle will be included in external tls connections. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "d4251366-de2d-4f5a-848b-d8f303f6a8e5", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "cf96c425-78a1-4e66-a560-3415a5faad6a", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "6a483ddc-747c-48af-8410-a8fb0d59cf83", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "12f07d32-5044-4fee-85a5-5ab013641c4d", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "d4509ea3-da67-4e63-bb58-3bf0ca8948b0", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "9c994a43-281d-4b42-b1dc-e0682b8b779f", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "7b5091fb-68b7-471d-91aa-f2f588cdfbf4", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "d36dde2b-caea-41e3-b351-39f2940a9f41", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "37709b9e-9ae2-4f07-8e61-4a5967e6c91a", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:14Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:14Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "377dfd3d-bfda-482e-be96-5fd6dff6b207", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:14Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "7a1e78f9-903a-479b-a0db-2ea00476c601", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "e33d88dd-b58b-479c-b48d-4f78309c7d8f", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "85c77231-c427-494b-9e59-5c8bd072b5c8", "error": "configmaps \"ds-pipelines-mariadb-tls-config-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "2882eabd-aa0b-4210-86cb-55016aced2c0", "error": "serviceaccounts \"ds-pipelines-mariadb-sa-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "dc28a605-8ca9-41c5-940d-7e40c1fb5b76", "error": "serviceaccounts \"ds-pipelines-mariadb-sa-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "cf9d3c04-b88a-4e74-b8aa-55b17389163c", "error": "serviceaccounts \"ds-pipelines-mariadb-sa-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "62368a7b-5242-468b-b10f-4cb914da3f4e", "error": "serviceaccounts \"ds-pipelines-mariadb-sa-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "26e87712-7941-400f-a8a1-119b2232abfc", "error": "serviceaccounts \"ds-pipelines-mariadb-sa-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "f6e630d7-5cc0-4fc6-87a2-f5b49b92224c", "error": "persistentvolumeclaims \"mariadb-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "439d3be9-1fde-4c8d-8ebe-c9a1063eadab", "error": "persistentvolumeclaims \"mariadb-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "0af3b804-105c-4eb4-801c-2531a7371c40", "error": "persistentvolumeclaims \"mariadb-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Applying mariaDB resources. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Publishing Ready Metrics {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO ScheduledWorkflowReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO DatabaseAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO PersistenceAgentReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO WorkflowControllerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO MLMDProxyReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO Ready {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:34:15Z INFO ObjectStoreAvailable {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z INFO APIServerReady {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:34:15Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"cy-e2e-automl-binary-522120"}, "namespace": "cy-e2e-automl-binary-522120", "name": "dspa", "reconcileID": "136b666f-4d1c-49c0-9081-28e486fdf249", "error": "persistentvolumeclaims \"mariadb-dspa\" is forbidden: unable to create new content in namespace cy-e2e-automl-binary-522120 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:34:15Z INFO No other DSPAs with PipelineStore 'kubernetes' found. Cleaning up webhook resources. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO Webhook resources cleanup complete. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T08:34:15Z INFO DSPA resources cleaned up. {"namespace": "cy-e2e-automl-binary-522120", "dspa_name": "dspa"} 2026-07-06T08:42:31Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:31Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:31Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:42:32Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:32Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:32Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:32Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:42:32Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:32Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:32Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:32Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:42:32Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:32Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:32Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:32Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:42:32Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:32Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:32Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:32Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:32Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:42:33Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:33Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:33Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:33Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:33Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:33Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:33Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:33Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:33Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:42:34Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:34Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:34Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:34Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:42:34Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:34Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:34Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:34Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:34Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:42:40Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:40Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:40Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:40Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:40Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:40Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:40Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:40Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:42:40Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:40Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:40Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:40Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:40Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:40Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:40Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:40Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:40Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:42:52Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:52Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:52Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:52Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:52Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:52Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:52Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:52Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:42:52Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:42:52Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:52Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:42:52Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:52Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:52Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:52Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:52Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:42:52Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:43:12Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:12Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:12Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:12Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:12Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:12Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:12Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:12Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:12Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:32Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:43:32Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:32Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:32Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:32Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:32Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:32Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:32Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:32Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:32Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:52Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:43:53Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:53Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:53Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:53Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:53Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:53Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:53Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:53Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:53Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:43:53Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:43:53Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:53Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:53Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:53Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:53Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:53Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:43:53Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:43:53Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:44:13Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:13Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:13Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:13Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:13Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:13Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:13Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:13Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:44:13Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:13Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:13Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:13Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:13Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:13Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:13Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:13Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:13Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:44:33Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:33Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:33Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:44:33Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:33Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:33Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:33Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:44:33Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:33Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:33Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:33Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:33Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:53Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:44:53Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:44:53Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:53Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:53Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:53Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:53Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:53Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:44:53Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:44:53Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:45:13Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:13Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:45:13Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:45:13Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:13Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:13Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:13Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:13Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:13Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:45:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:45:33Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:33Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:45:33Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:45:33Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:45:33Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:33Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:33Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:33Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:33Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:53Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:45:53Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:45:53Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:45:53Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:53Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:53Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:45:53Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:45:53Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:53Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:45:53Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:13Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:14Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:14Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:14Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:14Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:14Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:14Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:14Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:14Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:46:14Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:14Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:14Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:14Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:14Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:14Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:46:14Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:14Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:46:14Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:46:14Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:46:34Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:46:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:34Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:34Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:46:34Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:46:34Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:46:54Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:46:54Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:54Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:46:54Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:46:54Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:46:54Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:54Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:54Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:46:54Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:47:14Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:14Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:14Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:14Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:14Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:47:14Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:47:14Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:14Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:14Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:47:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:47:34Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:34Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:47:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:47:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:34Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:34Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:34Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:47:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:47:54Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:47:54Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:54Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:54Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:54Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:54Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:47:54Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:47:54Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:47:54Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:48:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:48:14Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:14Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:48:14Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:48:14Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:14Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:48:14Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:14Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:14Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:14Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:48:34Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:34Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:48:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:48:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:34Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:34Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:34Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:48:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:48:54Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:48:54Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:48:54Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:48:54Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:54Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:54Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:54Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:54Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:48:54Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:14Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:15Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:15Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:15Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:15Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:15Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:15Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:15Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:49:15Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:15Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:15Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:15Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:15Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:15Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:15Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:15Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:15Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:15Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:15Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:49:35Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:35Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:35Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:35Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:49:35Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:35Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:35Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:35Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:49:35Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:35Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:35Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:35Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:35Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:35Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:55Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:49:55Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:49:55Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:55Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:55Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:55Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:55Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:55Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:49:55Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:49:55Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:50:15Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:15Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:15Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:15Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:15Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:15Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:15Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:15Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:15Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:33Z INFO No other DSPAs with PipelineStore 'kubernetes' found. Cleaning up webhook resources. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T08:50:33Z INFO Webhook resources cleanup complete. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T08:50:33Z INFO DSPA resources cleaned up. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:39Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:39Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:39Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:50:40Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:40Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:40Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:40Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:50:40Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:40Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:40Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:40Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:50:40Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:40Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:40Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:40Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:50:40Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:40Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:40Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:40Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:40Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:40Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:41Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:50:41Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:41Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:41Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:41Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:41Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:41Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:41Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:41Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:41Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:42Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:50:42Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:42Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:42Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:42Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:42Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:42Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:42Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:42Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:42Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:50:42Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:42Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:42Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:42Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:42Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:42Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:42Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:42Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:42Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:50Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:50:50Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:50Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:50Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:50Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:50Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:50Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:50Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:50Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:50Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:51Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:50:51Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:50:51Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:51Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:51Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:51Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:51Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:51Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:50:51Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:50:51Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:02Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:51:02Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:51:02Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:51:02Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:02Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:02Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:02Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:02Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:02Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:51:02Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:51:02Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:02Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:02Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:02Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:51:02Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:02Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:51:02Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:51:02Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:02Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:22Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:51:22Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:22Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:51:22Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:22Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:22Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:51:22Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:51:22Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:22Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:22Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:42Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:51:42Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:51:42Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:51:42Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:42Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:42Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:51:42Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:51:42Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:42Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:51:42Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:02Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:52:02Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:02Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:02Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:02Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:52:02Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:52:02Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:02Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:02Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:02Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:52:22Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:52:22Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:22Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:22Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:22Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:52:22Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:52:22Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:22Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:52:22Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:22Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:42Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:52:43Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:52:43Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:52:43Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:43Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:43Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:52:43Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:52:43Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:43Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:52:43Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:53:03Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:03Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:03Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:03Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:53:03Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:03Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:03Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:03Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:53:03Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:03Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:03Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:03Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:03Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:03Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:23Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:53:23Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:23Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:23Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:23Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:23Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:23Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:23Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:23Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:23Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:43Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:53:43Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:43Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:43Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:43Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:43Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:43Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:43Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:43Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:43Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:53:43Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:53:43Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:43Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:43Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:43Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:43Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:43Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:53:43Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:53:43Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:03Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:54:03Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:54:03Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:54:03Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:03Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:03Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:03Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:03Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:54:03Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:03Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:54:03Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:03Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:54:03Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:54:03Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:03Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:03Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:03Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:54:03Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:03Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:23Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:54:23Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:23Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:54:23Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:54:23Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:54:23Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:23Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:23Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:23Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:23Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:43Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:54:44Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:54:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:44Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:44Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:54:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:54:44Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:54:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:54:44Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:04Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:55:04Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:04Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:04Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:55:04Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:04Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:04Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:04Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:04Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:55:04Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:55:24Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:55:24Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:24Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:55:24Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:24Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:24Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:55:24Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:24Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:24Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:24Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:55:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:55:44Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:55:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:44Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:55:44Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:44Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:55:44Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:55:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:55:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:04Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:56:04Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:04Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:56:04Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:04Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:04Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:04Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:04Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:56:04Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:04Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:56:24Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:56:24Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:24Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:24Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:24Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:24Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:56:24Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:56:24Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:56:24Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:24Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:56:44Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:56:44Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:44Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:56:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:56:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:44Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:56:44Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:57:04Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:57:04Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:04Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:57:04Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:04Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:57:04Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:57:04Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:04Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:04Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:04Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:24Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:57:24Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:24Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:57:24Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:24Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:24Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:24Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:57:24Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:57:24Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:24Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:44Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:45Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:45Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:45Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:45Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:45Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:45Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:45Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:45Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:45Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:57:45Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:45Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:45Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:57:45Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:57:45Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:45Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:45Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:45Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:45Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:57:45Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:57:45Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:05Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:05Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:05Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:05Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:05Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:05Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:05Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:05Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:05Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:05Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:25Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:25Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:25Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:25Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:25Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:25Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:25Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Applying mariaDB resources. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Finished applying Database Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Finished applying storage Resources {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Performing Database Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:25Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:25Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:25Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:25Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:25Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:25Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:33Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-custom-pip-prj-150747 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:33Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:33Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:33Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:33Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:33Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-custom-pip-prj-150747 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:33Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:33Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:33Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:33Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-custom-pip-prj-150747 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:33Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:33Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:33Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:33Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-custom-pip-prj-150747 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:33Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:33Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:33Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:33Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:33Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-custom-pip-prj-150747 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:34Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-custom-pip-prj-150747 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:34Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-custom-pip-prj-150747 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:34Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-custom-pip-prj-150747 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:34Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-custom-pip-prj-150747, this bundle will be included in external tls connections. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:34Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:34Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z INFO APIServerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO MLMDProxyReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO Ready {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:58:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z INFO DatabaseAvailable {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:58:34Z ERROR Encountered error when updating the DSPA status {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa", "error": "Operation cannot be fulfilled on datasciencepipelinesapplications.datasciencepipelinesapplications.opendatahub.io \"dspa\": the object has been modified; please apply your changes to the latest version and try again"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).updateStatus /workspace/controllers/dspipeline_controller.go:595 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:453 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:58:34Z INFO No other DSPAs with PipelineStore 'kubernetes' found. Cleaning up webhook resources. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T08:58:34Z INFO Webhook resources cleanup complete. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T08:58:34Z INFO DSPA resources cleaned up. {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:58:34Z INFO DataSciencePipelinesApplication.datasciencepipelinesapplications.opendatahub.io "dspa" not found {"namespace": "test-dsp-custom-pip-prj-150747", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:59:34Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:34Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:34Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:34Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:59:34Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:34Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:34Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:34Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:59:34Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:34Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:34Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:34Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:34Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:59:35Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:35Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:35Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:35Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:35Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:35Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:35Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:35Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:59:35Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:35Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:35Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:35Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:35Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:35Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:35Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:35Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:35Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:37Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:59:37Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:37Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:37Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:37Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:37Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:37Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:37Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:37Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:37Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:59:37Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:37Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:37Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:37Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:37Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:37Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:37Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:37Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:37Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:41Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:59:41Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:41Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:41Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:41Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:41Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:41Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:41Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:41Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:41Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:59:41Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:41Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:41Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:41Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:41Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:41Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:41Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:41Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:41Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:59:52Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:52Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:52Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:52Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:52Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:52Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:52Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:52Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T08:59:52Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T08:59:52Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:52Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:52Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:52Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:52Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:52Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T08:59:52Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T08:59:52Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:00:12Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:12Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:12Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:12Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:12Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:12Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:12Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:12Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:12Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:32Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:00:32Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:32Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:32Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:32Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:32Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:32Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:32Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:32Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:32Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:00:52Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:52Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:52Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:52Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:52Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:52Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:52Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:52Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:52Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:00:53Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:53Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:53Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:53Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:53Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:53Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:53Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:53Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:53Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:00:53Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:00:53Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:53Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:53Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:53Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:53Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:53Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:00:53Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:00:53Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:01:13Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:13Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:13Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:13Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:13Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:13Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:13Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:13Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:13Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:01:33Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:33Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:33Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:33Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:33Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:33Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:33Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:33Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:33Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:01:53Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:53Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:53Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:53Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:01:53Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:53Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:53Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:53Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:01:53Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:01:53Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:53Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:01:53Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:01:53Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:02:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:02:13Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:13Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:13Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:13Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:13Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:13Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:02:13Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:02:13Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:02:13Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:02:33Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:33Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:02:33Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:33Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:33Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:33Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:33Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:33Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:02:33Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:02:53Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:02:54Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:02:54Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:02:54Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:54Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:02:54Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:02:54Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:54Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:54Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:02:54Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:03:14Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:14Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:03:14Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:14Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:14Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:14Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:03:14Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:03:14Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:14Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:03:34Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:34Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:03:34Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:34Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:34Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:34Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:03:34Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:34Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:34Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:03:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:03:54Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:03:54Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:03:54Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:54Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:54Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:54Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:03:54Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:03:54Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:03:54Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:04:14Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:14Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:04:14Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:14Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:04:14Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:04:14Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:14Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:14Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:14Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:04:34Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:34Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:04:34Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:34Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:34Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:34Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:04:34Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:04:34Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:34Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:04:54Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:04:54Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:04:54Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:04:54Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:54Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:54Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:54Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:54Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:04:54Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:05:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:05:14Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:14Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:14Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:05:14Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:14Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:14Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:05:14Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:05:14Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:14Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:05:34Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:34Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:05:34Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:05:34Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:34Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:34Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:34Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:34Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:34Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:05:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:05:55Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:55Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:55Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:05:55Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:05:55Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:55Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:55Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:55Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:05:55Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:05:55Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:05:55Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:05:55Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:55Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:55Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:55Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:55Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:05:55Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:05:55Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:06:15Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:15Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:15Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:15Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:06:15Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:15Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:15Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:06:15Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:15Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:15Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:15Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:06:15Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:15Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:15Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:15Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:15Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:06:35Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:35Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:35Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:35Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:35Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:35Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:35Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:35Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:35Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:55Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:06:55Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:06:55Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:55Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:55Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:55Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:55Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:06:55Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:55Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:06:55Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:15Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:15Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:15Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:15Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:15Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:15Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:15Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:15Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:15Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:35Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:35Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:35Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:35Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:35Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:35Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:35Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:35Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:35Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:35Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:35Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:35Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:35Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:35Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:35Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:35Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:35Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:35Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:35Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:35Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:38Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:38Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:38Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:38Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:38Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:38Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:38Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:38Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:38Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:38Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:38Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:38Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:38Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:38Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:38Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:38Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:38Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:38Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:38Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:38Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:38Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:38Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:39Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:39Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:39Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:39Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:39Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:39Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:39Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:39Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:39Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:39Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:39Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:39Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:39Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:39Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:39Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:39Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:39Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:39Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:39Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:39Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:39Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:39Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:39Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:39Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:39Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:39Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:39Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:39Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:39Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:39Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:39Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:40Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:40Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:40Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:40Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:40Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:40Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:40Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:40Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:40Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:40Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:41Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:41Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:41Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:41Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:41Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:41Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:41Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:41Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:41Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:41Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:41Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:41Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:41Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:41Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:41Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:41Z INFO No other DSPAs with PipelineStore 'kubernetes' found. Cleaning up webhook resources. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T09:07:41Z INFO Webhook resources cleanup complete. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T09:07:41Z INFO DSPA resources cleaned up. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:46Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:46Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:46Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:46Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:46Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:46Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:46Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:46Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:46Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:46Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:46Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:46Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:46Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:46Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:46Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:46Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:46Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:46Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:47Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:47Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:47Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:47Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:47Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:47Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:47Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:47Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:47Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:47Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:47Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:47Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:47Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:48Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:48Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:48Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:48Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:48Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:48Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:48Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:48Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:48Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:48Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:48Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:48Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:48Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:48Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:48Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:48Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:48Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:48Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:48Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:51Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:51Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:51Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:51Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:51Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:51Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:51Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:51Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:51Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:51Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:07:52Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:07:52Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:52Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:52Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:52Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:52Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:52Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:07:52Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:07:52Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:03Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:08:03Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:08:03Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:03Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:08:03Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:03Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:03Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:03Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:03Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:08:03Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:08:03Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:03Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:03Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:08:03Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:03Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:03Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:08:03Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:08:03Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:03Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:23Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:08:23Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:23Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:08:23Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:08:23Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:23Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:23Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:08:23Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:23Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:23Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:43Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:08:43Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:08:43Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:08:43Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:43Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:43Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:43Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:08:43Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:08:43Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:08:43Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:03Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:09:03Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:03Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:09:03Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:09:03Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:03Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:03Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:03Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:03Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:03Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:09:23Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:09:23Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:23Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:23Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:23Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:09:23Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:09:23Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:23Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:23Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:09:23Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:43Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:09:44Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:09:44Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:09:44Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:44Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:44Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:44Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:44Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:09:44Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:09:44Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:10:04Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:10:04Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:04Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:04Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:04Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:10:04Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:10:04Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:10:04Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:04Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:04Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:24Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:10:24Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:24Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:10:24Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:24Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:24Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:10:24Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:10:24Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:24Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:24Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:10:44Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:10:44Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:10:44Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:44Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:44Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:44Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:44Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:10:44Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:10:44Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:11:04Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:11:04Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:04Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:04Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:04Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:04Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:11:04Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:04Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:04Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:11:04Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:11:24Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:11:24Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:24Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:24Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:24Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:11:24Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:11:24Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:11:24Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:24Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:24Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:11:44Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:11:44Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:44Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:11:44Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:44Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:44Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:11:44Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:11:44Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:11:44Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:04Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:12:04Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:04Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:04Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:04Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:04Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:04Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:04Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:04Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:04Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:12:04Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:04Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:04Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:04Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:04Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:04Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:04Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:04Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:04Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:24Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:12:25Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:25Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:25Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:25Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:25Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:25Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:25Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:25Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:25Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:45Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:12:45Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:45Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:45Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:45Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:45Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:45Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:45Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:45Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:45Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:12:45Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:12:45Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:45Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:45Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:45Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:45Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:45Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:12:45Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:12:45Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:05Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:13:05Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:05Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:05Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:05Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:05Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:05Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:05Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:05Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:05Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:13:25Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:25Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:25Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:13:25Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:25Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:25Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:25Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:13:25Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:25Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:25Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:25Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:25Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:45Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:13:45Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:13:45Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:45Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:45Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:45Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:45Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:45Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:13:45Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:13:45Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:05Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:14:05Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:05Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:14:05Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:14:05Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:05Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:14:05Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:05Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:05Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:05Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:14:26Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:26Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:14:26Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:14:26Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:26Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:26Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:26Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:26Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:14:26Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:46Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:14:46Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:14:46Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:46Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:46Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:14:46Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:14:46Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:46Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:14:46Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:14:46Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:06Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:06Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:06Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:06Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:06Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:06Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:06Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:06Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:06Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:06Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:26Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO Applying mariaDB resources. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO Finished applying Database Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO Finished applying storage Resources {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO Performing Database Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO Performing Object Storage Health Check {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:26Z INFO Object Storage Health Check Failed {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:26Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:26Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:26Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:26Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:26Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:26Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:26Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:26Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:38Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:38Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:38Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:38Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:38Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:38Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-pipelines-prj-643148, this bundle will be included in external tls connections. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:38Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-pipelines-prj-643148 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:38Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:38Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:38Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:38Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:38Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:38Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:38Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:38Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:38Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:38Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:38Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:38Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:38Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:39Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:39Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:39Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:39Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:39Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:39Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:39Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:39Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:39Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:39Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:39Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:39Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:39Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:39Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:39Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:39Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:39Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:39Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:39Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:39Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:39Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:39Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:39Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:39Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:39Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:40Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:40Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:40Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:40Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:40Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:40Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:40Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:40Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:40Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:40Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:40Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:40Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:40Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:40Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:40Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:40Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:40Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:15:40Z INFO Publishing Ready Metrics {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:40Z INFO DatabaseAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO ObjectStoreAvailable {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO WorkflowControllerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO MLMDProxyReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO Ready {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:40Z INFO APIServerReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:15:40Z INFO PersistenceAgentReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO ScheduledWorkflowReady {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:15:40Z INFO No other DSPAs with PipelineStore 'kubernetes' found. Cleaning up webhook resources. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T09:15:40Z INFO Webhook resources cleanup complete. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T09:15:40Z INFO DSPA resources cleaned up. {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:15:40Z INFO DataSciencePipelinesApplication.datasciencepipelinesapplications.opendatahub.io "dspa" not found {"namespace": "test-pipelines-prj-643148", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:16:40Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:40Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:40Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:40Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:16:40Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:40Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:40Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:40Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:16:40Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:40Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:40Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:16:40Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:40Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:40Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:40Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:40Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:41Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:16:41Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:41Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:41Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:41Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:41Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:41Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:41Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:41Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:41Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:43Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:16:43Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:43Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:43Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:43Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:43Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:43Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:43Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:43Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:43Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:16:43Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:43Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:43Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:43Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:43Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:43Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:43Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:43Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:43Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:16:52Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:52Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:52Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:52Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:52Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:52Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:52Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:52Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:52Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:16:53Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:16:53Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:53Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:16:53Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:53Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:53Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:53Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:53Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:16:53Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:04Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:17:04Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:04Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:04Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:04Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:04Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:04Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:04Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:04Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:04Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:17:04Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:04Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:04Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:04Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:04Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:04Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:04Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:04Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:04Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:24Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:17:24Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:24Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:24Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:24Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:24Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:24Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:24Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:24Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:24Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:17:44Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:17:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:17:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:17:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:04Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:18:04Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:04Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:18:04Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:18:04Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:04Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:04Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:04Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:18:04Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:04Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:24Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:18:24Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:24Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:18:24Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:18:24Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:24Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:24Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:24Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:24Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:24Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:18:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:18:44Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:18:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:18:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:18:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:18:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:18:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:04Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:19:04Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:04Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:04Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:04Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:04Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:04Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:04Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:04Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:04Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:19:04Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:04Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:04Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:04Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:04Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:04Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:04Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:04Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:04Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:24Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:19:25Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:25Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:25Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:19:25Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:25Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:25Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:19:25Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:25Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:25Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:19:25Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:25Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:25Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:25Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:25Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:25Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:19:45Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:45Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:45Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:45Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:19:45Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:45Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:45Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:45Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:19:45Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:19:45Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:45Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:19:45Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:19:45Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:05Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:20:05Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:05Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:05Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:05Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:05Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:05Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:05Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:05Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:05Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:20:05Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:05Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:05Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:05Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:05Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:05Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:05Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:05Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:05Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:05Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:06Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:06Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:06Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:06Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:06Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:06Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:06Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:06Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:06Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:20:06Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:06Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:06Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:06Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:06Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:06Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:06Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:06Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:06Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:06Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:06Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:26Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:20:26Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:26Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:26Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:26Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:26Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:26Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:26Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:26Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:26Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:46Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:20:46Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:20:46Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:46Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:46Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:20:46Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:46Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:46Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:46Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:20:46Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:21:06Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:21:06Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:06Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:06Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:06Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:06Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:06Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:06Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:21:06Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:21:06Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:21:26Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:21:26Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:26Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:21:26Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:21:26Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:26Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:21:26Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:26Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:26Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:26Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:46Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:21:46Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:21:46Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:46Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:21:46Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:46Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:21:46Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:21:46Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:46Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:21:46Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:06Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:22:06Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:06Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:06Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:06Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:06Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:06Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:06Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:06Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:06Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:26Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:22:26Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:26Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:26Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:26Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:26Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:26Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:26Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:26Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:26Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:46Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:22:46Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:46Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:46Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:46Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:46Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:46Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:46Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:46Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:46Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:46Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:22:47Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:47Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:47Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:47Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:47Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:47Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:47Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:47Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:47Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:22:47Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:22:47Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:47Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:47Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:47Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:47Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:47Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:22:47Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:22:47Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:23:07Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:07Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:07Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:07Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:23:07Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:07Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:07Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:07Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:23:07Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:07Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:07Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:07Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:07Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:27Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:23:27Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:27Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:27Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:27Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:27Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:27Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:27Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:27Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:27Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:47Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:23:47Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:23:47Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:47Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:47Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:47Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:47Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:47Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:23:47Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:23:47Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:07Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:07Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:07Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:07Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:07Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:07Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:07Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:07Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:07Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:07Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:27Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:27Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:27Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:27Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:27Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:27Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:27Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:27Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:27Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:27Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:27Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:27Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:27Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:27Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:27Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:27Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:27Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:27Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-schedule-prj-115465 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-schedule-prj-115465 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-schedule-prj-115465 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-schedule-prj-115465 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-schedule-prj-115465 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-schedule-prj-115465 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-schedule-prj-115465 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-schedule-prj-115465 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:45Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-schedule-prj-115465 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:45Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:45Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:45Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:45Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:45Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-schedule-prj-115465 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:45Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:45Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:45Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:45Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:45Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-schedule-prj-115465 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:45Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:45Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:45Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:45Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:45Z ERROR Encountered error when updating the DSPA status {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "Operation cannot be fulfilled on datasciencepipelinesapplications.datasciencepipelinesapplications.opendatahub.io \"dspa\": the object has been modified; please apply your changes to the latest version and try again"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).updateStatus /workspace/controllers/dspipeline_controller.go:595 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:453 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:45Z INFO No other DSPAs with PipelineStore 'kubernetes' found. Cleaning up webhook resources. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T09:24:45Z INFO Webhook resources cleanup complete. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T09:24:45Z INFO DSPA resources cleaned up. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:51Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:51Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:51Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:51Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:51Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:51Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:51Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:51Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:51Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:51Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:51Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:51Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:51Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:51Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:51Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:51Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:51Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:51Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:52Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:52Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:52Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:52Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:52Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:52Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:52Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:52Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:52Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:52Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:53Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:53Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:53Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:53Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:53Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:53Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:53Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:53Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:53Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:53Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:24:53Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:24:53Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:53Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:53Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:53Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:24:53Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:53Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:53Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:24:53Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:01Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:25:01Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:01Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:01Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:01Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:01Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:01Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:01Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:01Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:01Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:02Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:25:02Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:02Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:02Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:02Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:02Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:02Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:02Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:02Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:02Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:25:13Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:13Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:13Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:13Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:13Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:13Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:13Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:13Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:25:13Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:13Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:13Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:13Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:13Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:13Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:13Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:13Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:13Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:25:14Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:14Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:14Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:14Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:14Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:14Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:14Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:14Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:25:14Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:14Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:14Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:14Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:14Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:14Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:14Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:14Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:14Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:25:34Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:34Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:34Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:34Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:34Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:25:54Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:25:54Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:54Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:54Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:54Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:54Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:25:54Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:54Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:25:54Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:26:14Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:14Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:26:14Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:26:14Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:14Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:26:14Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:14Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:14Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:14Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:26:34Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:34Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:26:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:26:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:34Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:34Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:34Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:26:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:26:54Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:26:54Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:26:54Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:54Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:54Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:26:54Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:54Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:54Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:26:54Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:27:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:27:14Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:14Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:27:14Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:14Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:14Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:14Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:14Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:27:14Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:14Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:27:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:27:34Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:34Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:27:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:27:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:34Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:34Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:34Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:27:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:27:54Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:27:54Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:27:54Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:54Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:27:54Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:27:54Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:54Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:54Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:27:54Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:28:14Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:14Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:28:14Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:14Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:28:14Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:28:14Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:14Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:14Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:14Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:34Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:28:34Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:34Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:28:34Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:34Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:28:34Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:34Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:34Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:34Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:34Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:28:55Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:28:55Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:28:55Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:55Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:55Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:55Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:28:55Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:28:55Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:28:55Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:28:55Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:29:15Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:15Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:29:15Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:29:15Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:15Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:29:15Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:15Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:15Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:15Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:29:35Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:35Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:29:35Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:35Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:35Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:35Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:35Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:35Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:29:35Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:29:55Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:29:55Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:29:55Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:29:55Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:29:55Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:55Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:55Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:55Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:55Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:29:55Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:30:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:30:15Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:15Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:15Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:15Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:30:15Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:15Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:15Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:15Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:30:15Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:30:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:30:35Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:35Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:30:35Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:35Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:35Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:35Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:35Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:30:35Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:35Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:30:55Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:30:55Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:30:55Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:30:55Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:55Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:55Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:30:55Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:55Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:55Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:30:55Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:31:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:31:15Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:15Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:31:15Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:15Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:15Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:31:15Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:15Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:15Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:15Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:31:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:31:35Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:35Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:31:35Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:35Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:35Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:35Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:31:35Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:35Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:35Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:31:55Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:31:55Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:31:55Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:31:55Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:55Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:55Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:31:55Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:31:55Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:55Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:31:55Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:15Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:15Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:15Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:15Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:15Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:15Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:15Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:15Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:16Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:16Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:16Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:16Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:16Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:16Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:16Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:16Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:16Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:16Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:16Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:16Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:36Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:36Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:36Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:36Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:36Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:36Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:36Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:36Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Applying mariaDB resources. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying Database Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Using externalStorage, bypassing object storage deployment. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Finished applying storage Resources {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Database Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Encountered error when processing custom ca bundle, Error: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Unable to connect to Database: error parsing CA Certificate, ensure provided certs are in valid PEM format {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Performing Object Storage Health Check {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z ERROR encountered error when processing custom ca bundle or proxy configuration {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "error parsing CA Certificate, ensure provided certs are in valid PEM format"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.init.func2 /workspace/controllers/storage.go:164 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).isObjectStorageAccessible /workspace/controllers/storage.go:255 github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:36Z INFO Object Storage Health Check Failed {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:36Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:36Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:36Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace test-dsp-schedule-prj-115465, this bundle will be included in external tls connections. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace test-dsp-schedule-prj-115465 because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:44Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:44Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:44Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:44Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:45Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:45Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:45Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:45Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:45Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:45Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:45Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:45Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:45Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:45Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:45Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:45Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:45Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:45Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:45Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:45Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:45Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:46Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:46Z ERROR Failed to extract or resolve DSPA parameters {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:315 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-06T09:32:46Z INFO Publishing Ready Metrics {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} 2026-07-06T09:32:46Z INFO APIServerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:46Z INFO PersistenceAgentReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:46Z INFO WorkflowControllerReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:46Z INFO MLMDProxyReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:46Z INFO Ready {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "False"} 2026-07-06T09:32:46Z INFO ObjectStoreAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:46Z INFO ScheduledWorkflowReady {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:46Z INFO DatabaseAvailable {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-06T09:32:46Z INFO No other DSPAs with PipelineStore 'kubernetes' found. Cleaning up webhook resources. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T09:32:46Z INFO Webhook resources cleanup complete. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-06T09:32:46Z INFO DSPA resources cleaned up. {"namespace": "test-dsp-schedule-prj-115465", "dspa_name": "dspa"} ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-7cc8755dc9-dhk4t.log ==================== W0706 08:11:32.940663 1 connect.go:86] Retrying connection to PostgreSQL (attempt 1/25): failed to connect to `user=catalog_user database=model_catalog`: 172.30.210.155:5432 (model-catalog-postgres): dial error: dial tcp 172.30.210.155:5432: connect: connection refused 172.30.210.155:5432 (model-catalog-postgres): dial error: dial tcp 172.30.210.155:5432: connect: connection refused W0706 08:11:33.943810 1 connect.go:86] Retrying connection to PostgreSQL (attempt 2/25): failed to connect to `user=catalog_user database=model_catalog`: 172.30.210.155:5432 (model-catalog-postgres): dial error: dial tcp 172.30.210.155:5432: connect: connection refused 172.30.210.155:5432 (model-catalog-postgres): dial error: dial tcp 172.30.210.155:5432: connect: connection refused W0706 08:11:35.946686 1 connect.go:86] Retrying connection to PostgreSQL (attempt 3/25): failed to connect to `user=catalog_user database=model_catalog`: 172.30.210.155:5432 (model-catalog-postgres): dial error: dial tcp 172.30.210.155:5432: connect: connection refused 172.30.210.155:5432 (model-catalog-postgres): dial error: dial tcp 172.30.210.155:5432: connect: connection refused W0706 08:11:38.951766 1 connect.go:86] Retrying connection to PostgreSQL (attempt 4/25): failed to connect to `user=catalog_user database=model_catalog`: 172.30.210.155:5432 (model-catalog-postgres): dial error: dial tcp 172.30.210.155:5432: connect: connection refused 172.30.210.155:5432 (model-catalog-postgres): dial error: dial tcp 172.30.210.155:5432: connect: connection refused W0706 08:11:42.957488 1 connect.go:86] Retrying connection to PostgreSQL (attempt 5/25): failed to connect to `user=catalog_user database=model_catalog`: 172.30.210.155:5432 (model-catalog-postgres): dial error: dial tcp 172.30.210.155:5432: connect: connection refused 172.30.210.155:5432 (model-catalog-postgres): dial error: dial tcp 172.30.210.155:5432: connect: connection refused W0706 08:11:47.964407 1 connect.go:86] Retrying connection to PostgreSQL (attempt 6/25): failed to connect to `user=catalog_user database=model_catalog`: 172.30.210.155:5432 (model-catalog-postgres): dial error: dial tcp 172.30.210.155:5432: connect: connection refused 172.30.210.155:5432 (model-catalog-postgres): dial error: dial tcp 172.30.210.155:5432: connect: connection refused I0706 08:11:53.978819 1 connect.go:94] Successfully connected to PostgreSQL database I0706 08:11:53.978903 1 catalog.go:121] Leader election configured: lock duration=1m0s, heartbeat=15s I0706 08:11:53.986463 1 service.go:130] Connecting to EmbedMD service... I0706 08:11:53.986504 1 service.go:137] Connected to EmbedMD service I0706 08:11:53.986509 1 repos.go:35] Getting types... I0706 08:11:53.986530 1 election.go:190] Attempting to acquire leadership... I0706 08:11:53.989562 1 election.go:226] Successfully acquired leadership lock: catalog-leader W0706 08:11:53.998041 1 service.go:186] Unable to get types: no types available: ERROR: relation "Type" does not exist (SQLSTATE 42P01) I0706 08:11:53.999099 1 service.go:162] Running migrations... I0706 08:11:54.197635 1 service.go:167] Migrations completed I0706 08:11:54.197668 1 service.go:169] Syncing types... I0706 08:11:54.261691 1 service.go:174] Syncing types completed W0706 08:11:54.261712 1 election.go:132] Leader callback 0 exited early I0706 08:11:54.998502 1 repos.go:35] Getting types... I0706 08:11:54.999789 1 repos.go:47] Types retrieved I0706 08:11:54.999818 1 repos.go:63] All required types validated successfully 2026/07/06 08:11:55 INFO initializing plugin plugin=mcp version=v1alpha1 basePath=/api/mcp_catalog/v1alpha1 2026/07/06 08:11:55 INFO initializing plugin plugin=model version=v1alpha1 basePath=/api/model_catalog/v1alpha1 I0706 08:11:55.000313 1 performance_metrics.go:154] Loading performance metrics data from [/shared-benchmark-data] I0706 08:11:55.006169 1 performance_metrics.go:239] Built model directory cache (72 models indexed) 2026/07/06 08:11:55 INFO all plugins initialized count=2 2026/07/06 08:11:55 INFO mounting plugin routes plugin=mcp 2026/07/06 08:11:55 INFO mounting plugin routes plugin=model 2026/07/06 08:11:55 INFO starting plugin plugin=mcp I0706 08:11:55.006339 1 base.go:49] Starting mcp plugin in read-only mode (standby) I0706 08:11:55.006358 1 loader.go:84] Initializing MCP loader - parsing configs W0706 08:11:55.007179 1 config.go:24] Configuration file /data/default-sources/sources.yaml uses deprecated 'catalogs' field. Please rename to 'model_catalogs'. I0706 08:11:55.007204 1 loader.go:149] reading MCP catalog config type yaml... I0706 08:11:55.007215 1 loader.go:165] loaded MCP source rh_mcp_servers of type yaml I0706 08:11:55.007219 1 loader.go:149] reading MCP catalog config type yaml... I0706 08:11:55.007223 1 loader.go:165] loaded MCP source rh_partner_mcp_servers of type yaml I0706 08:11:55.007226 1 loader.go:149] reading MCP catalog config type yaml... I0706 08:11:55.007229 1 loader.go:165] loaded MCP source community_mcp_servers of type yaml I0706 08:11:55.007328 1 loader.go:92] MCP loader config parsing complete I0706 08:11:55.007345 1 base.go:64] mcp plugin read-only mode initialized 2026/07/06 08:11:55 INFO starting plugin plugin=model I0706 08:11:55.007358 1 base.go:49] Starting model plugin in read-only mode (standby) W0706 08:11:55.007750 1 config.go:24] Configuration file /data/default-sources/sources.yaml uses deprecated 'catalogs' field. Please rename to 'model_catalogs'. I0706 08:11:55.007765 1 loader.go:204] reading config type yaml... I0706 08:11:55.007776 1 loader.go:222] loaded source redhat_ai_models of type yaml I0706 08:11:55.007780 1 loader.go:204] reading config type yaml... I0706 08:11:55.007784 1 loader.go:222] loaded source redhat_ai_validated_models of type yaml I0706 08:11:55.007787 1 loader.go:204] reading config type yaml... I0706 08:11:55.007790 1 loader.go:222] loaded source other_models of type yaml I0706 08:11:55.007932 1 base.go:64] model plugin read-only mode initialized I0706 08:11:55.008084 1 catalog.go:195] Became leader - notifying plugins I0706 08:11:55.008086 1 catalog.go:176] Catalog API server listening on 0.0.0.0:8080 2026/07/06 08:11:55 INFO plugin becoming leader plugin=model I0706 08:11:55.008178 1 base.go:97] model plugin becoming leader 2026/07/06 08:11:55 INFO plugin becoming leader plugin=mcp I0706 08:11:55.008210 1 base.go:97] mcp plugin becoming leader I0706 08:11:55.008249 1 loader.go:101] MCP loader performing leader operations I0706 08:11:55.008266 1 loader.go:192] Loading MCP servers from source: Red Hat MCP Servers (id: rh_mcp_servers) I0706 08:11:55.011184 1 loader.go:379] Reading models from yaml source redhat_ai_models I0706 08:11:55.021911 1 loader.go:379] Reading models from yaml source redhat_ai_validated_models I0706 08:11:55.118348 1 loader.go:379] Reading models from yaml source other_models I0706 08:11:55.123266 1 base.go:112] model plugin leader mode active I0706 08:11:55.123272 1 loader.go:285] Loading model redhat_ai_models:RedHatAI/granite-3.3-8b-instruct with 1 artifact(s) I0706 08:11:55.150400 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Apertus-8B-Instruct-2509-FP8-dynamic with 1 artifact(s) I0706 08:11:55.329921 1 loader.go:192] Loading MCP servers from source: Red Hat Partner MCP Servers (id: rh_partner_mcp_servers) I0706 08:11:55.434478 1 performance_metrics.go:273] Loaded 117 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Apertus-8B-Instruct-2509-FP8-dynamic I0706 08:11:55.434543 1 loader.go:285] Loading model redhat_ai_models:RedHatAI/granite-4.0-h-small with 1 artifact(s) I0706 08:11:55.452705 1 loader.go:285] Loading model other_models:RedHatAI/Qwen3-Embedding-8B with 1 artifact(s) I0706 08:11:55.470634 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/DeepSeek-R1-0528-quantized.w4a16 with 1 artifact(s) I0706 08:11:55.632698 1 performance_metrics.go:273] Loaded 55 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/DeepSeek-R1-0528-quantized.w4a16 I0706 08:11:55.632736 1 loader.go:285] Loading model redhat_ai_models:RedHatAI/granite-4.0-h-tiny with 1 artifact(s) I0706 08:11:55.646379 1 loader.go:285] Loading model other_models:RedHatAI/all-MiniLM-L6-v2 with 1 artifact(s) I0706 08:11:55.662485 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Devstral-Small-2-24B-Instruct-2512 with 1 artifact(s) I0706 08:11:55.960607 1 performance_metrics.go:273] Loaded 138 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Devstral-Small-2-24B-Instruct-2512 I0706 08:11:55.960662 1 loader.go:285] Loading model redhat_ai_models:granite-7b-redhat-lab with 1 artifact(s) I0706 08:11:55.970452 1 loader.go:192] Loading MCP servers from source: Community MCP Servers (id: community_mcp_servers) I0706 08:11:55.976248 1 loader.go:285] Loading model other_models:RedHatAI/embeddinggemma-300m with 1 artifact(s) I0706 08:11:56.037014 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Kimi-K2-Instruct-quantized.w4a16 with 1 artifact(s) I0706 08:11:56.095278 1 performance_metrics.go:273] Loaded 22 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Kimi-K2-Instruct-quantized.w4a16 I0706 08:11:56.095310 1 loader.go:285] Loading model redhat_ai_models:granite-8b-starter-v1 with 1 artifact(s) I0706 08:11:56.133355 1 loader.go:285] Loading model other_models:RedHatAI/granite-embedding-english-r2 with 1 artifact(s) I0706 08:11:56.135340 1 loader.go:115] MCP loader leader operations complete I0706 08:11:56.135360 1 base.go:112] mcp plugin leader mode active I0706 08:11:56.148894 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.1-8B-Instruct with 1 artifact(s) I0706 08:11:56.327289 1 performance_metrics.go:273] Loaded 107 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.1-8B-Instruct I0706 08:11:56.327334 1 loader.go:285] Loading model redhat_ai_models:granite-8b-lab-v1 with 1 artifact(s) I0706 08:11:56.340918 1 loader.go:285] Loading model other_models:RedHatAI/nomic-embed-text-v1.5 with 1 artifact(s) I0706 08:11:56.340966 1 loader.go:420] other_models: loaded 5 models I0706 08:11:56.355522 1 loader.go:431] other_models: cleaned up 0 models I0706 08:11:56.359161 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.1-Nemotron-70B-Instruct-HF with 1 artifact(s) I0706 08:11:56.439925 1 performance_metrics.go:273] Loaded 30 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.1-Nemotron-70B-Instruct-HF I0706 08:11:56.440053 1 loader.go:285] Loading model redhat_ai_models:granite-3.1-8b-lab-v1 with 1 artifact(s) I0706 08:11:56.454079 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.1-Nemotron-70B-Instruct-HF-FP8-dynamic with 1 artifact(s) I0706 08:11:56.523842 1 performance_metrics.go:273] Loaded 35 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.1-Nemotron-70B-Instruct-HF-FP8-dynamic I0706 08:11:56.523877 1 loader.go:285] Loading model redhat_ai_models:granite-8b-code-instruct with 1 artifact(s) I0706 08:11:56.537100 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct with 1 artifact(s) I0706 08:11:56.608172 1 performance_metrics.go:273] Loaded 35 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct I0706 08:11:56.608207 1 loader.go:285] Loading model redhat_ai_models:granite-8b-code-base with 1 artifact(s) I0706 08:11:56.621629 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic with 1 artifact(s) I0706 08:11:56.694431 1 performance_metrics.go:273] Loaded 36 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic I0706 08:11:56.694469 1 loader.go:285] Loading model redhat_ai_models:prometheus-8x7b-v2-0 with 1 artifact(s) I0706 08:11:56.694528 1 loader.go:420] redhat_ai_models: loaded 10 models I0706 08:11:56.710262 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct-quantized.w4a16 with 1 artifact(s) I0706 08:11:56.711341 1 loader.go:431] redhat_ai_models: cleaned up 0 models I0706 08:11:56.808497 1 performance_metrics.go:273] Loaded 30 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct-quantized.w4a16 I0706 08:11:56.808537 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct-quantized.w8a8 with 1 artifact(s) I0706 08:11:56.893994 1 performance_metrics.go:273] Loaded 43 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-3.3-70B-Instruct-quantized.w8a8 I0706 08:11:56.894032 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Llama-4-Maverick-17B-128E-Instruct with 1 artifact(s) I0706 08:11:56.934954 1 performance_metrics.go:273] Loaded 3 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-4-Maverick-17B-128E-Instruct I0706 08:11:56.934997 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Llama-4-Maverick-17B-128E-Instruct-FP8 with 1 artifact(s) I0706 08:11:56.994713 1 performance_metrics.go:273] Loaded 18 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-4-Maverick-17B-128E-Instruct-FP8 I0706 08:11:56.994748 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Llama-4-Scout-17B-16E-Instruct with 1 artifact(s) I0706 08:11:57.046870 1 performance_metrics.go:273] Loaded 15 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-4-Scout-17B-16E-Instruct I0706 08:11:57.046922 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic with 1 artifact(s) I0706 08:11:57.122830 1 performance_metrics.go:273] Loaded 40 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic I0706 08:11:57.122884 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16 with 1 artifact(s) I0706 08:11:57.181550 1 performance_metrics.go:273] Loaded 16 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16 I0706 08:11:57.181630 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic with 1 artifact(s) I0706 08:11:57.230132 1 performance_metrics.go:273] Loaded 3 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic I0706 08:11:57.230183 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w4a16 with 1 artifact(s) I0706 08:11:57.254292 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 with 1 artifact(s) I0706 08:11:57.277892 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/MiniMax-M2.5 with 1 artifact(s) I0706 08:11:57.358660 1 performance_metrics.go:273] Loaded 41 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/MiniMax-M2.5 I0706 08:11:57.358697 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Ministral-3-14B-Instruct-2512 with 1 artifact(s) I0706 08:11:57.630293 1 performance_metrics.go:273] Loaded 130 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Ministral-3-14B-Instruct-2512 I0706 08:11:57.630342 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Ministral-3-3B-Instruct-2512 with 1 artifact(s) I0706 08:11:57.845173 1 performance_metrics.go:273] Loaded 149 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Ministral-3-3B-Instruct-2512 I0706 08:11:57.845218 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Large-3-675B-Instruct-2512 with 1 artifact(s) I0706 08:11:57.895640 1 performance_metrics.go:273] Loaded 18 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Large-3-675B-Instruct-2512 I0706 08:11:57.895672 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Large-3-675B-Instruct-2512-NVFP4 with 1 artifact(s) I0706 08:11:57.968555 1 performance_metrics.go:273] Loaded 42 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Large-3-675B-Instruct-2512-NVFP4 I0706 08:11:57.968588 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-24B-Instruct-2501 with 1 artifact(s) I0706 08:11:58.084890 1 performance_metrics.go:273] Loaded 81 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-24B-Instruct-2501 I0706 08:11:58.084930 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-24B-Instruct-2501-FP8-dynamic with 1 artifact(s) I0706 08:11:58.132092 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w4a16 with 1 artifact(s) I0706 08:11:58.149938 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w8a8 with 1 artifact(s) I0706 08:11:58.168800 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503 with 1 artifact(s) I0706 08:11:58.340069 1 performance_metrics.go:273] Loaded 105 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503 I0706 08:11:58.340113 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic with 1 artifact(s) I0706 08:11:58.534097 1 performance_metrics.go:273] Loaded 112 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic I0706 08:11:58.534186 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16 with 1 artifact(s) I0706 08:11:58.730351 1 performance_metrics.go:273] Loaded 104 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16 I0706 08:11:58.730441 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w8a8 with 1 artifact(s) I0706 08:11:58.932706 1 performance_metrics.go:273] Loaded 123 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w8a8 I0706 08:11:58.932851 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-4-119B-2603 with 1 artifact(s) I0706 08:11:59.053962 1 performance_metrics.go:273] Loaded 40 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-4-119B-2603 I0706 08:11:59.054078 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mistral-Small-4-119B-2603-NVFP4 with 1 artifact(s) I0706 08:11:59.233274 1 performance_metrics.go:273] Loaded 91 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mistral-Small-4-119B-2603-NVFP4 I0706 08:11:59.233343 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Mixtral-8x7B-Instruct-v0.1 with 1 artifact(s) I0706 08:11:59.350846 1 performance_metrics.go:273] Loaded 59 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Mixtral-8x7B-Instruct-v0.1 I0706 08:11:59.350891 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 with 1 artifact(s) I0706 08:11:59.560426 1 performance_metrics.go:273] Loaded 126 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 I0706 08:11:59.560558 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 with 1 artifact(s) I0706 08:11:59.728173 1 performance_metrics.go:273] Loaded 80 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 I0706 08:11:59.728218 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 with 1 artifact(s) I0706 08:11:59.869141 1 performance_metrics.go:273] Loaded 79 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 I0706 08:11:59.869182 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 with 1 artifact(s) I0706 08:11:59.954253 1 performance_metrics.go:273] Loaded 31 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 I0706 08:11:59.954296 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-Nano-9B-v2-FP8-dynamic with 1 artifact(s) I0706 08:12:00.048135 1 performance_metrics.go:273] Loaded 46 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/NVIDIA-Nemotron-Nano-9B-v2-FP8-dynamic I0706 08:12:00.048189 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Phi-4-mini-instruct-FP8-dynamic with 1 artifact(s) I0706 08:12:00.251575 1 performance_metrics.go:273] Loaded 110 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Phi-4-mini-instruct-FP8-dynamic I0706 08:12:00.251618 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Phi-4-reasoning-FP8-dynamic with 1 artifact(s) I0706 08:12:00.387345 1 performance_metrics.go:273] Loaded 75 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Phi-4-reasoning-FP8-dynamic I0706 08:12:00.387389 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct with 1 artifact(s) I0706 08:12:00.574722 1 performance_metrics.go:273] Loaded 102 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct I0706 08:12:00.574762 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct-FP8-dynamic with 1 artifact(s) I0706 08:12:00.762376 1 performance_metrics.go:273] Loaded 110 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct-FP8-dynamic I0706 08:12:00.762416 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct-quantized.w4a16 with 1 artifact(s) I0706 08:12:01.003997 1 performance_metrics.go:273] Loaded 87 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct-quantized.w4a16 I0706 08:12:01.004055 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct-quantized.w8a8 with 1 artifact(s) I0706 08:12:01.236781 1 performance_metrics.go:273] Loaded 106 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen2.5-7B-Instruct-quantized.w8a8 I0706 08:12:01.236852 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Qwen3-8B-FP8-dynamic with 1 artifact(s) I0706 08:12:01.475993 1 performance_metrics.go:273] Loaded 155 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3-8B-FP8-dynamic I0706 08:12:01.476033 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Qwen3-Coder-480B-A35B-Instruct-FP8 with 1 artifact(s) I0706 08:12:01.542353 1 performance_metrics.go:273] Loaded 13 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3-Coder-480B-A35B-Instruct-FP8 I0706 08:12:01.542393 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Qwen3-Coder-Next-NVFP4 with 1 artifact(s) I0706 08:12:01.752937 1 performance_metrics.go:273] Loaded 119 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3-Coder-Next-NVFP4 I0706 08:12:01.753004 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Qwen3-Next-80B-A3B-Instruct-quantized.w4a16 with 1 artifact(s) I0706 08:12:01.972733 1 performance_metrics.go:273] Loaded 133 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3-Next-80B-A3B-Instruct-quantized.w4a16 I0706 08:12:01.972777 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Qwen3-VL-235B-A22B-Instruct-NVFP4 with 1 artifact(s) I0706 08:12:02.095377 1 performance_metrics.go:273] Loaded 79 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3-VL-235B-A22B-Instruct-NVFP4 I0706 08:12:02.095425 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Qwen3.5-122B-A10B-FP8-dynamic with 1 artifact(s) I0706 08:12:02.218525 1 performance_metrics.go:273] Loaded 42 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3.5-122B-A10B-FP8-dynamic I0706 08:12:02.218568 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Qwen3.5-35B-A3B-FP8-dynamic with 1 artifact(s) I0706 08:12:02.431237 1 performance_metrics.go:273] Loaded 108 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3.5-35B-A3B-FP8-dynamic I0706 08:12:02.431304 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic with 1 artifact(s) I0706 08:12:02.513726 1 performance_metrics.go:273] Loaded 25 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic I0706 08:12:02.513768 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/Voxtral-Mini-3B-2507-FP8-dynamic with 1 artifact(s) I0706 08:12:02.547431 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/gemma-2-9b-it with 1 artifact(s) I0706 08:12:02.567833 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/gemma-2-9b-it-FP8 with 1 artifact(s) I0706 08:12:02.589258 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/gemma-3n-E4B-it-FP8-dynamic with 1 artifact(s) I0706 08:12:02.757010 1 performance_metrics.go:273] Loaded 81 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gemma-3n-E4B-it-FP8-dynamic I0706 08:12:02.757072 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/gpt-oss-120b with 1 artifact(s) I0706 08:12:03.035994 1 performance_metrics.go:273] Loaded 148 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gpt-oss-120b I0706 08:12:03.036034 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/gpt-oss-120b-essential with 1 artifact(s) I0706 08:12:03.337911 1 performance_metrics.go:273] Loaded 148 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gpt-oss-120b-essential I0706 08:12:03.337971 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/gpt-oss-20b with 1 artifact(s) I0706 08:12:03.644028 1 performance_metrics.go:273] Loaded 177 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gpt-oss-20b I0706 08:12:03.644073 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/gpt-oss-20b-essential with 1 artifact(s) I0706 08:12:03.951474 1 performance_metrics.go:273] Loaded 177 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/gpt-oss-20b-essential I0706 08:12:03.951557 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-base-quantized.w4a16 with 1 artifact(s) I0706 08:12:04.009177 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct with 2 artifact(s) I0706 08:12:04.194030 1 performance_metrics.go:273] Loaded 95 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct I0706 08:12:04.194094 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct-FP8-dynamic with 1 artifact(s) I0706 08:12:04.260633 1 performance_metrics.go:273] Loaded 6 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct-FP8-dynamic I0706 08:12:04.260680 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct-quantized.w4a16 with 1 artifact(s) I0706 08:12:04.342781 1 performance_metrics.go:273] Loaded 4 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct-quantized.w4a16 I0706 08:12:04.342842 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct-quantized.w8a8 with 1 artifact(s) I0706 08:12:04.420206 1 performance_metrics.go:273] Loaded 1 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/granite-3.1-8b-instruct-quantized.w8a8 I0706 08:12:04.420242 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/granite-4.0-h-small-FP8-dynamic with 1 artifact(s) I0706 08:12:04.529938 1 performance_metrics.go:273] Loaded 42 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/granite-4.0-h-small-FP8-dynamic I0706 08:12:04.529992 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/granite-4.0-h-tiny-FP8-dynamic with 1 artifact(s) I0706 08:12:04.656100 1 performance_metrics.go:273] Loaded 46 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/granite-4.0-h-tiny-FP8-dynamic I0706 08:12:04.656151 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/phi-4 with 1 artifact(s) I0706 08:12:04.789996 1 performance_metrics.go:273] Loaded 55 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/phi-4 I0706 08:12:04.790050 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/phi-4-FP8-dynamic with 1 artifact(s) I0706 08:12:04.956131 1 performance_metrics.go:273] Loaded 54 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/phi-4-FP8-dynamic I0706 08:12:04.956176 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/phi-4-quantized.w4a16 with 1 artifact(s) I0706 08:12:05.071700 1 performance_metrics.go:273] Loaded 52 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/phi-4-quantized.w4a16 I0706 08:12:05.071748 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/phi-4-quantized.w8a8 with 1 artifact(s) I0706 08:12:05.185718 1 performance_metrics.go:273] Loaded 60 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/phi-4-quantized.w8a8 I0706 08:12:05.185761 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/sarvam-105b-FP8-dynamic with 1 artifact(s) I0706 08:12:05.353131 1 performance_metrics.go:273] Loaded 61 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/sarvam-105b-FP8-dynamic I0706 08:12:05.353182 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/sarvam-30b-FP8-dynamic with 1 artifact(s) I0706 08:12:05.543093 1 performance_metrics.go:273] Loaded 105 performance metrics artifacts for model redhat_ai_validated_models:RedHatAI/sarvam-30b-FP8-dynamic I0706 08:12:05.543153 1 loader.go:285] Loading model redhat_ai_validated_models:RedHatAI/whisper-large-v3-turbo-quantized.w4a16 with 1 artifact(s) I0706 08:12:05.543237 1 loader.go:420] redhat_ai_validated_models: loaded 76 models I0706 08:12:05.592219 1 loader.go:431] redhat_ai_validated_models: cleaned up 0 models 2026/07/06 08:30:31 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:52522 - 200 472B in 440.907µs 2026/07/06 08:30:31 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:52518 - 200 420B in 6.953136ms 2026/07/06 08:30:31 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:52518 - 200 9130B in 4.933587ms 2026/07/06 08:30:31 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:52518 - 200 84064B in 7.836885ms 2026/07/06 08:30:31 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:52522 - 200 141549B in 10.40124ms 2026/07/06 08:30:31 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:52528 - 200 47545B in 20.599676ms 2026/07/06 08:30:32 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52522 - 200 41125B in 116.067966ms 2026/07/06 08:30:32 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52518 - 200 62309B in 113.895675ms 2026/07/06 08:30:32 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52552 - 200 20726B in 110.767289ms 2026/07/06 08:30:32 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgemma-2-9b-it/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52558 - 200 56B in 19.15045ms 2026/07/06 08:30:49 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic HTTP/1.1" from 127.0.0.1:52522 - 200 16070B in 2.688841ms 2026/07/06 08:30:49 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts HTTP/1.1" from 127.0.0.1:52518 - 200 32975B in 9.858643ms 2026/07/06 08:30:49 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?filterQuery=variant_group_id.string_value%3D%22c359df52-82cb-4d43-be32-6dafea427522%22&pageSize=10&source=redhat_ai_validated_models HTTP/1.1" from 127.0.0.1:52522 - 200 16124B in 2.818115ms 2026/07/06 08:30:49 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts HTTP/1.1" from 127.0.0.1:52518 - 200 32975B in 9.596028ms 2026/07/06 08:30:49 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&pageSize=20&recommendations=true&targetRPS=1 HTTP/1.1" from 127.0.0.1:52518 - 200 41124B in 38.027421ms 2026/07/06 08:30:50 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90&pageSize=20&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52518 - 200 41124B in 30.329675ms 2026/07/06 08:31:25 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:52522 - 200 472B in 178.567µs 2026/07/06 08:31:25 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:52518 - 200 420B in 1.833781ms 2026/07/06 08:31:25 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:52518 - 200 9130B in 1.945806ms 2026/07/06 08:31:25 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:52518 - 200 47545B in 5.652685ms 2026/07/06 08:31:25 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:52522 - 200 141549B in 8.554621ms 2026/07/06 08:31:25 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:37944 - 200 84064B in 23.297035ms 2026/07/06 08:31:25 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgemma-2-9b-it/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52518 - 200 56B in 5.540263ms 2026/07/06 08:31:25 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52522 - 200 20726B in 90.757704ms 2026/07/06 08:31:25 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52518 - 200 41125B in 97.325941ms 2026/07/06 08:31:25 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:37952 - 200 62309B in 125.761728ms 2026/07/06 08:31:38 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic HTTP/1.1" from 127.0.0.1:52522 - 200 16070B in 2.295402ms 2026/07/06 08:31:38 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts HTTP/1.1" from 127.0.0.1:52518 - 200 32975B in 8.491646ms 2026/07/06 08:31:38 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts HTTP/1.1" from 127.0.0.1:52518 - 200 32975B in 9.091649ms 2026/07/06 08:31:39 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?filterQuery=variant_group_id.string_value%3D%22c359df52-82cb-4d43-be32-6dafea427522%22&pageSize=10&source=redhat_ai_validated_models HTTP/1.1" from 127.0.0.1:52518 - 200 16124B in 3.069246ms 2026/07/06 08:31:39 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&pageSize=20&recommendations=true&targetRPS=1 HTTP/1.1" from 127.0.0.1:52518 - 200 41124B in 31.552096ms 2026/07/06 08:31:39 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90&pageSize=20&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52518 - 200 41124B in 34.912723ms 2026/07/06 08:36:08 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:48040 - 200 420B in 2.563401ms 2026/07/06 08:36:14 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:48040 - 200 420B in 1.720748ms 2026/07/06 08:36:28 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:48040 - 200 420B in 2.126554ms 2026/07/06 08:36:51 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:51408 - 200 472B in 312.529µs 2026/07/06 08:36:51 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:48040 - 200 9130B in 3.158725ms 2026/07/06 08:36:51 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:51410 - 200 420B in 3.219579ms 2026/07/06 08:36:52 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:48040 - 200 141549B in 8.369674ms 2026/07/06 08:36:52 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:48040 - 200 47545B in 4.584497ms 2026/07/06 08:36:52 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:51408 - 200 84064B in 6.910559ms 2026/07/06 08:36:52 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgemma-2-9b-it/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51420 - 200 56B in 52.424481ms 2026/07/06 08:36:52 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51408 - 200 20726B in 61.169998ms 2026/07/06 08:36:52 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:48040 - 200 41125B in 73.57734ms 2026/07/06 08:36:52 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51418 - 200 62309B in 96.913426ms 2026/07/06 08:37:11 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:51408 - 200 472B in 85.675µs 2026/07/06 08:37:11 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:51420 - 200 9130B in 2.461332ms 2026/07/06 08:37:11 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:39414 - 200 420B in 2.880266ms 2026/07/06 08:37:12 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:51420 - 200 141549B in 8.39688ms 2026/07/06 08:37:12 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:51420 - 200 47545B in 3.595432ms 2026/07/06 08:37:12 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:51420 - 200 84064B in 5.390808ms 2026/07/06 08:37:12 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51408 - 200 20726B in 31.348071ms 2026/07/06 08:37:12 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51420 - 200 62309B in 52.40554ms 2026/07/06 08:37:12 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgemma-2-9b-it/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51420 - 200 56B in 7.287658ms 2026/07/06 08:37:12 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:39424 - 200 41125B in 69.602678ms 2026/07/06 08:37:43 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:51420 - 200 420B in 1.712418ms 2026/07/06 08:37:43 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:51420 - 200 472B in 54.693µs 2026/07/06 08:37:43 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:51420 - 200 9130B in 1.994216ms 2026/07/06 08:37:43 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:51420 - 200 47545B in 4.055735ms 2026/07/06 08:37:43 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:51408 - 200 141549B in 6.243993ms 2026/07/06 08:37:43 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:51420 - 200 84064B in 4.872774ms 2026/07/06 08:37:46 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgemma-2-9b-it/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51408 - 200 56B in 5.464914ms 2026/07/06 08:37:46 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51420 - 200 20726B in 34.227425ms 2026/07/06 08:37:46 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51408 - 200 62309B in 57.831853ms 2026/07/06 08:37:46 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:55076 - 200 41125B in 83.971852ms 2026/07/06 08:37:58 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?filterQuery=artifacts.use_case.string_value%3D%27chatbot%27+AND+artifacts.ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=artifacts.ttft_p90.double_value&pageSize=10&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51408 - 200 292781B in 1.44543933s 2026/07/06 08:37:59 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FMinistral-3-3B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51420 - 200 29027B in 53.096017ms 2026/07/06 08:37:59 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgpt-oss-20b/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51408 - 200 52497B in 71.187599ms 2026/07/06 08:37:59 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FMinistral-3-14B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51420 - 200 59712B in 97.675569ms 2026/07/06 08:37:59 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgpt-oss-20b-essential/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51408 - 200 52679B in 110.262385ms 2026/07/06 08:37:59 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:56222 - 200 41125B in 184.690383ms 2026/07/06 08:37:59 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:56224 - 200 62309B in 194.982549ms 2026/07/06 08:37:59 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fsarvam-30b-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51420 - 200 24427B in 184.529159ms 2026/07/06 08:37:59 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FQwen2.5-7B-Instruct-quantized.w8a8/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51408 - 200 35997B in 188.943568ms 2026/07/06 08:37:59 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FQwen3-8B-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:56224 - 200 37135B in 192.102572ms 2026/07/06 08:37:59 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FLlama-3.1-8B-Instruct/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:56222 - 200 55242B in 185.152879ms 2026/07/06 08:38:15 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:51420 - 200 472B in 126.709µs 2026/07/06 08:38:15 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:51408 - 200 420B in 2.676923ms 2026/07/06 08:38:15 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:51408 - 200 9130B in 3.998472ms 2026/07/06 08:38:16 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:51408 - 200 141549B in 9.412464ms 2026/07/06 08:38:16 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:51420 - 200 47545B in 4.968275ms 2026/07/06 08:38:16 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:51408 - 200 84064B in 6.408039ms 2026/07/06 08:38:16 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgemma-2-9b-it/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51420 - 200 56B in 8.735051ms 2026/07/06 08:38:16 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51408 - 200 41125B in 73.321303ms 2026/07/06 08:38:16 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:33162 - 200 20726B in 83.426226ms 2026/07/06 08:38:16 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:33176 - 200 62309B in 89.40649ms 2026/07/06 08:38:30 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?filterQuery=artifacts.use_case.string_value%3D%27chatbot%27+AND+artifacts.ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=artifacts.ttft_p90.double_value&pageSize=10&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51408 - 200 292781B in 1.651645067s 2026/07/06 08:38:30 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FMinistral-3-14B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51408 - 200 59712B in 38.63638ms 2026/07/06 08:38:30 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgpt-oss-20b-essential/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51420 - 200 52679B in 260.408632ms 2026/07/06 08:38:30 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:43358 - 200 62309B in 258.281745ms 2026/07/06 08:38:30 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:43366 - 200 41125B in 343.84871ms 2026/07/06 08:38:30 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgpt-oss-20b/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51408 - 200 52497B in 341.243969ms 2026/07/06 08:38:30 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fsarvam-30b-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:43378 - 200 24427B in 284.52719ms 2026/07/06 08:38:30 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FMinistral-3-3B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:43352 - 200 29027B in 364.131721ms 2026/07/06 08:38:31 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FQwen2.5-7B-Instruct-quantized.w8a8/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:43358 - 200 35997B in 171.462468ms 2026/07/06 08:38:31 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FQwen3-8B-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:43366 - 200 37135B in 124.636111ms 2026/07/06 08:38:31 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FLlama-3.1-8B-Instruct/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:51420 - 200 55242B in 109.133724ms 2026/07/06 08:38:41 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:43366 - 200 472B in 82.461µs 2026/07/06 08:38:41 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:43358 - 200 420B in 2.364945ms 2026/07/06 08:38:41 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:43358 - 200 9130B in 3.670994ms 2026/07/06 08:38:41 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:43358 - 200 84064B in 6.137243ms 2026/07/06 08:38:41 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:43358 - 200 47545B in 3.865957ms 2026/07/06 08:38:41 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:43358 - 200 141549B in 6.449303ms 2026/07/06 08:38:43 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgemma-2-9b-it/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:53748 - 200 56B in 45.559319ms 2026/07/06 08:38:43 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:43366 - 200 20726B in 51.538176ms 2026/07/06 08:38:43 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:43358 - 200 41125B in 59.133539ms 2026/07/06 08:38:43 "GET http://rh-ai.apps.ods-qe-psi-07.osp.rh-ods.com/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:53738 - 200 62309B in 80.526292ms W0706 08:11:33.036385 1 deprecated.go:66] ==== Removed Flag Warning ====================== logtostderr is removed in the k8s upstream and has no effect any more. =============================================== I0706 08:11:33.036511 1 kube-rbac-proxy.go:553] Reading config file: /etc/kube-rbac-proxy/config-file.yaml I0706 08:11:33.037675 1 kube-rbac-proxy.go:247] Valid token audiences: I0706 08:11:33.038708 1 kube-rbac-proxy.go:370] Reading certificate files I0706 08:11:33.039142 1 kube-rbac-proxy.go:418] Starting TCP socket on 0.0.0.0:8443 I0706 08:11:33.039176 1 kube-rbac-proxy.go:466] Starting TCP socket on 0.0.0.0:8888 I0706 08:11:33.039515 1 kube-rbac-proxy.go:425] Listening securely on 0.0.0.0:8443 I0706 08:11:33.039525 1 kube-rbac-proxy.go:473] Listening securely on 0.0.0.0:8888 for proxy endpoints ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-postgres-686d85b578-v4bv9.log ==================== The files belonging to this database system will be owned by user "postgres". This user must also own the server process. The database cluster will be initialized with locale "en_US.utf8". The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english". Data page checksums are disabled. fixing permissions on existing directory /var/lib/pgsql/data/userdata ... ok creating subdirectories ... ok selecting dynamic shared memory implementation ... posix selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting default time zone ... UTC creating configuration files ... ok running bootstrap script ... ok performing post-bootstrap initialization ... ok syncing data to disk ... ok Success. You can now start the database server using: pg_ctl -D /var/lib/pgsql/data/userdata -l logfile start initdb: warning: enabling "trust" authentication for local connections initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. waiting for server to start....2026-07-06 08:11:38.465 UTC [30] LOG: redirecting log output to logging collector process 2026-07-06 08:11:38.465 UTC [30] HINT: Future log output will appear in directory "log". done server started /var/run/postgresql:5432 - accepting connections => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ... ALTER ROLE waiting for server to shut down.... done server stopped Starting server... 2026-07-06 08:11:38.875 UTC [1] LOG: redirecting log output to logging collector process 2026-07-06 08:11:38.875 UTC [1] HINT: Future log output will appear in directory "log".