==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== 143m Warning FailedMount pod/spark-pi-d95qth7dpamc7b5mqnfg-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d95qth7dpamc7b5mqnfg-2c69009f37a2cbde-driver-podspec-conf-map" not found 143m Warning FailedMount pod/spark-pi-d95qth7dpamc7b5mqnfg-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-fd13da9f37a2dd64-conf-map" not found 141m Warning FailedMount pod/scheduled-spark-pi-d95qttndpamc7b5mqng0-1783344916000395417-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-f692aa9f37a414c1-conf-map" not found 91m Warning FailedToUpdateEndpointSlices service/ogx-k8s-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-service: failed to update ogx-k8s-operator-controller-manager-metrics-service-zx9lr EndpointSlice for Service redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-service: endpointslices.discovery.k8s.io "ogx-k8s-operator-controller-manager-metrics-service-zx9lr" not found 91m Warning FailedCreate replicaset/ogx-k8s-operator-controller-manager-67866ffc79 Error creating: pods "ogx-k8s-operator-controller-manager-67866ffc79-" is forbidden: error looking up service account redhat-ods-applications/ogx-k8s-operator-controller-manager: serviceaccount "ogx-k8s-operator-controller-manager" not found 11m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 73m Warning Deleting modelregistry/postgres-registry-1783348916931 Custom Resource postgres-registry-1783348916931 is being deleted from the namespace rhoai-model-registries 73m Warning Deleting modelregistry/postgres-registry-1783348916931 Custom Resource postgres-registry-1783348916931 is being deleted from the namespace rhoai-model-registries 72m Warning Deleting modelregistry/default-db-registry-1783348916931 Custom Resource default-db-registry-1783348916931 is being deleted from the namespace rhoai-model-registries 72m Warning Deleting modelregistry/default-db-registry-1783348916931 Custom Resource default-db-registry-1783348916931 is being deleted from the namespace rhoai-model-registries 67m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 74m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 74m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 74m Warning FailedMount pod/e2e-test-registry-d9cbcd957-97swx MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "e2e-test-registry-kube-rbac-proxy-config" not found 72m Warning Unhealthy pod/default-db-registry-1783348916931-55b9c575f7-klv7x Startup probe failed: Get "http://10.129.2.39:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 72m Warning Unhealthy pod/default-db-registry-1783348916931-55b9c575f7-klv7x Startup probe failed: HTTP probe failed with statuscode: 503 72m Warning Unhealthy pod/mysql-db-registry-1783348916931-7465475dfc-6x4ll Readiness probe errored: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 71m Warning Unhealthy pod/postgres-db-registry-1783348916931-5df959fcdd-pm9kq Liveness probe failed: /var/run/postgresql:5432 - rejecting connections 71m Warning Unhealthy pod/default-db-registry-1783348916931-postgres-748f77b47d-w2kn5 Readiness probe errored: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 79m Warning FailedMount pod/jupyter-nb-ldap-2dadmin1-0 MountVolume.SetUp failed for volume "kube-rbac-proxy-tls-certificates" : secret "jupyter-nb-ldap-2dadmin1-kube-rbac-proxy-tls" not found 79m Warning FailedMount notebook/jupyter-nb-ldap-2dadmin1 Reissued from pod/jupyter-nb-ldap-2dadmin1-0: MountVolume.SetUp failed for volume "kube-rbac-proxy-tls-certificates" : secret "jupyter-nb-ldap-2dadmin1-kube-rbac-proxy-tls" not found 78m Warning Unhealthy pod/jupyter-nb-ldap-2dadmin1-0 Readiness probe failed: Get "http://10.129.2.32:8888/notebook/rhods-notebooks/jupyter-nb-ldap-2dadmin1/api": read tcp 10.129.2.2:59408->10.129.2.32:8888: read: connection reset by peer 78m Warning Unhealthy pod/jupyter-nb-ldap-2dadmin1-0 Readiness probe failed: Get "https://10.129.2.32:8444/healthz": dial tcp 10.129.2.32:8444: connect: connection refused 78m Warning Unhealthy notebook/jupyter-nb-ldap-2dadmin1 Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Readiness probe failed: Get "http://10.129.2.32:8888/notebook/rhods-notebooks/jupyter-nb-ldap-2dadmin1/api": read tcp 10.129.2.2:59408->10.129.2.32:8888: read: connection reset by peer 78m Warning Unhealthy notebook/jupyter-nb-ldap-2dadmin1 Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Readiness probe failed: Get "https://10.129.2.32:8444/healthz": dial tcp 10.129.2.32:8444: connect: connection refused {"level":"error","ts":"2026-07-06T15:53:50Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T15:54:38Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T15:55:27Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T15:55:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T15:56:42Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-07-06T15:57:20Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.19.18 Kubernetes Version: v1.32.9 --- Namespaces --- namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.0 Authorino Operator 1.4.0 authorino-operator.v1.3.1 Succeeded cert-manager-operator.v1.19.0 cert-manager Operator for Red Hat OpenShift 1.19.0 cert-manager-operator.v1.18.1 Succeeded custom-metrics-autoscaler.v2.19.0-1 Custom Metrics Autoscaler 2.19.0-1 custom-metrics-autoscaler.v2.18.1-2 Succeeded dns-operator.v1.4.0 DNS Operator 1.4.0 dns-operator.v1.3.1 Succeeded kueue-operator.v1.3.1 Red Hat build of Kueue 1.3.1 Succeeded limitador-operator.v1.4.0 Limitador Operator 1.4.0 limitador-operator.v1.3.1 Succeeded rhcl-operator.v1.4.0 Red Hat Connectivity Link 1.4.0 rhcl-operator.v1.3.4 Succeeded rhods-operator.3.5.0-ea.2 Red Hat OpenShift AI 3.5.0-ea.2 Succeeded servicemeshoperator3.v3.3.4 Red Hat OpenShift Service Mesh 3 3.3.4 servicemeshoperator3.v3.1.0 Succeeded --- CSVs not in Succeeded phase --- (all CSVs report phase Succeeded) --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aipipelines: managementState=Managed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Unmanaged llamastackoperator: managementState=n/a mlflowoperator: managementState=Managed modelregistry: managementState=Managed ogx: managementState=Removed ray: managementState=Managed sparkoperator: managementState=Managed trainer: managementState=Managed trainingoperator: managementState=Managed trustyai: managementState=Managed workbenches: managementState=Managed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= AIPipelinesReady: status=True reason= message= DashboardReady: status=True reason= message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=True reason= message= MLflowOperatorReady: status=True reason= message= MaaSPrerequisitesAvailable: status=True reason=MaaSPrerequisitesMet message=MaaS gateway annotations and Authorino TLS are correctly configured ModelControllerReady: status=True reason= message= ModelRegistryReady: status=True reason= message= ModelsAsServiceReady: status=True reason=Reconciled message=MaaS platform manifests applied and maas-api deployment is available OGXReady: status=False reason=Removed message=Component ManagementState is set to Removed RayReady: status=True reason= message= SparkOperatorReady: status=True reason= message= TrainerReady: status=True reason= message= TrainingOperatorReady: status=True reason= message= TrustyAIReady: status=True reason= message= WorkbenchesReady: status=True reason= message= --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 7d2h Ready 2026-06-29T13:43:18Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=True reason=Ready message=Monitoring stack is initialized Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in DSCI CR OpenTelemetryCollectorAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Metrics not configured in DSCI CR Traces not configured in DSCI CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in DSCI CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR PersesAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR --- Component CRs (Ready) --- NAME READY REASON URL dashboard.components.platform.opendatahub.io/default-dashboard True NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON kueue.components.platform.opendatahub.io/default-kueue True NAME READY REASON mlflowoperator.components.platform.opendatahub.io/default-mlflowoperator True NAME READY REASON URI modelcontroller.components.platform.opendatahub.io/default-modelcontroller True NAME READY REASON modelregistry.components.platform.opendatahub.io/default-modelregistry True NAME READY REASON modelsasservice.components.platform.opendatahub.io/default-modelsasservice True NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON sparkoperator.components.platform.opendatahub.io/default-sparkoperator True NAME READY REASON trainer.components.platform.opendatahub.io/default-trainer True NAME READY REASON trainingoperator.components.platform.opendatahub.io/default-trainingoperator True NAME READY REASON trustyai.components.platform.opendatahub.io/default-trustyai True NAME READY REASON workbenches.components.platform.opendatahub.io/default-workbenches True --- Deployments not fully ready (ODH/RHOAI namespaces) --- (all deployments report ready replicas == desired) --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- (all pods Running or Succeeded) --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 143m Warning FailedMount pod/spark-pi-d95qth7dpamc7b5mqnfg-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d95qth7dpamc7b5mqnfg-2c69009f37a2cbde-driver-podspec-conf-map" not found 143m Warning FailedMount pod/spark-pi-d95qth7dpamc7b5mqnfg-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-fd13da9f37a2dd64-conf-map" not found 141m Warning FailedMount pod/scheduled-spark-pi-d95qttndpamc7b5mqng0-1783344916000395417-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-f692aa9f37a414c1-conf-map" not found 91m Warning FailedToUpdateEndpointSlices service/ogx-k8s-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-service: failed to update ogx-k8s-operator-controller-manager-metrics-service-zx9lr EndpointSlice for Service redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-service: endpointslices.discovery.k8s.io "ogx-k8s-operator-controller-manager-metrics-service-zx9lr" not found 91m Warning FailedCreate replicaset/ogx-k8s-operator-controller-manager-67866ffc79 Error creating: pods "ogx-k8s-operator-controller-manager-67866ffc79-" is forbidden: error looking up service account redhat-ods-applications/ogx-k8s-operator-controller-manager: serviceaccount "ogx-k8s-operator-controller-manager" not found 11m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor ServiceMonitor controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- (no events) --- namespace/rhoai-model-registries --- LAST SEEN TYPE REASON OBJECT MESSAGE 67m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 73m Warning Deleting modelregistry/postgres-registry-1783348916931 Custom Resource postgres-registry-1783348916931 is being deleted from the namespace rhoai-model-registries 73m Warning Deleting modelregistry/postgres-registry-1783348916931 Custom Resource postgres-registry-1783348916931 is being deleted from the namespace rhoai-model-registries 72m Warning Deleting modelregistry/default-db-registry-1783348916931 Custom Resource default-db-registry-1783348916931 is being deleted from the namespace rhoai-model-registries 72m Warning Deleting modelregistry/default-db-registry-1783348916931 Custom Resource default-db-registry-1783348916931 is being deleted from the namespace rhoai-model-registries 67m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 74m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 74m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 74m Warning FailedMount pod/e2e-test-registry-d9cbcd957-97swx MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "e2e-test-registry-kube-rbac-proxy-config" not found 72m Warning Unhealthy pod/default-db-registry-1783348916931-55b9c575f7-klv7x Startup probe failed: Get "http://10.129.2.39:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 72m Warning Unhealthy pod/default-db-registry-1783348916931-55b9c575f7-klv7x Startup probe failed: HTTP probe failed with statuscode: 503 72m Warning Unhealthy pod/mysql-db-registry-1783348916931-7465475dfc-6x4ll Readiness probe errored: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 71m Warning Unhealthy pod/postgres-db-registry-1783348916931-5df959fcdd-pm9kq Liveness probe failed: /var/run/postgresql:5432 - rejecting connections 71m Warning Unhealthy pod/default-db-registry-1783348916931-postgres-748f77b47d-w2kn5 Readiness probe errored: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 --- namespace/rhods-notebooks --- LAST SEEN TYPE REASON OBJECT MESSAGE 79m Warning FailedMount pod/jupyter-nb-ldap-2dadmin1-0 MountVolume.SetUp failed for volume "kube-rbac-proxy-tls-certificates" : secret "jupyter-nb-ldap-2dadmin1-kube-rbac-proxy-tls" not found 79m Warning FailedMount notebook/jupyter-nb-ldap-2dadmin1 Reissued from pod/jupyter-nb-ldap-2dadmin1-0: MountVolume.SetUp failed for volume "kube-rbac-proxy-tls-certificates" : secret "jupyter-nb-ldap-2dadmin1-kube-rbac-proxy-tls" not found 78m Warning Unhealthy pod/jupyter-nb-ldap-2dadmin1-0 Readiness probe failed: Get "http://10.129.2.32:8888/notebook/rhods-notebooks/jupyter-nb-ldap-2dadmin1/api": read tcp 10.129.2.2:59408->10.129.2.32:8888: read: connection reset by peer 78m Warning Unhealthy pod/jupyter-nb-ldap-2dadmin1-0 Readiness probe failed: Get "https://10.129.2.32:8444/healthz": dial tcp 10.129.2.32:8444: connect: connection refused 78m Warning Unhealthy notebook/jupyter-nb-ldap-2dadmin1 Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Readiness probe failed: Get "http://10.129.2.32:8888/notebook/rhods-notebooks/jupyter-nb-ldap-2dadmin1/api": read tcp 10.129.2.2:59408->10.129.2.32:8888: read: connection reset by peer 78m Warning Unhealthy notebook/jupyter-nb-ldap-2dadmin1 Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Readiness probe failed: Get "https://10.129.2.32:8444/healthz": dial tcp 10.129.2.32:8444: connect: connection refused --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-07-06T15:53:16Z) --- redhat-ods-operator/rhods-operator-6f6bb89fdf-59gvk --- {"level":"error","ts":"2026-07-06T15:53:50Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T15:54:38Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T15:55:27Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T15:55:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T15:56:42Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-6f6bb89fdf-59gvk.log ==================== {"level":"error","ts":"2026-07-06T15:53:50Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"error","ts":"2026-07-06T15:54:38Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-06T15:54:52Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b"} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b"} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T15:54:57Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:54:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"d0d92d67-6ed3-4628-ba6b-fa4bbff57b6b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T15:54:59Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"error","ts":"2026-07-06T15:55:27Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T15:55:57Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:55:57Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm","msg":"EnvoyFilter exposes internal implementation details that may change at any time. Prefer other APIs if possible, and exercise extreme caution, especially around upgrades.","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:55:57Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"308dd038-a281-4168-a49c-acd4678aa60a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-06T15:55:59Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"d6938e89-887a-47dc-bb41-f0bcf5c86dd4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T15:56:12Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"d059af61-b5f1-479c-9fb2-568c1752e4e5","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"d059af61-b5f1-479c-9fb2-568c1752e4e5","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:56:12Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"37c69fad-4d41-4ede-8071-46dd8dc4b41c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"error","ts":"2026-07-06T15:56:42Z","logger":"controller-runtime.cache.UnhandledError","msg":"Failed to watch","reflector":"pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:289","type":"operator.authorino.kuadrant.io/v1beta1, Kind=Authorino","error":"authorinos.operator.authorino.kuadrant.io is forbidden: User \"system:serviceaccount:redhat-ods-operator:redhat-ods-operator-controller-manager\" cannot watch resource \"authorinos\" in API group \"operator.authorino.kuadrant.io\" at the cluster scope","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:204\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:370\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/client-go@v0.35.3/tools/cache/reflector.go:368\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/cachi2/output/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:72"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T15:57:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"afac99fc-5df0-423a-a8de-c1959ac3af46","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-06T15:57:12Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects.createDashboardRedirects","msg":"Creating dashboard redirect resources","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c","dashboardRouteName":"rhods-dashboard","namespace":"redhat-ods-applications","currentSubdomain":"rh-ai"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:12Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"5839ea7e-30c1-45d9-ab4e-f0cb23b5a23c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c0cf7c03-9630-4120-8206-e0c686420ed4","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-06T15:57:13Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"aa2828d5-ba68-4bff-83f9-ebf7df86fc86","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"aa2828d5-ba68-4bff-83f9-ebf7df86fc86","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-06T15:57:13Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"77ef4911-f89c-473d-bf92-c05cf8b30dde","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} ==================== OPERATOR POD LOG: rhods-operator-6f6bb89fdf-f4zpw.log ==================== ==================== OPERATOR POD LOG: rhods-operator-6f6bb89fdf-kr2pd.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-6bd46884d4-bzk82.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/data-science-pipelines-operator-controller-manager-7fc7f76tgzn2.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/feast-operator-controller-manager-744655f689-4vl7d.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-controller-manager-6c8cb86d7f-n6mh9.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kubeflow-trainer-controller-manager-776d978f97-x8nc4.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kubeflow-training-operator-85b5d44979-xj4vv.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kuberay-operator-6b449f854b-ffqj6.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/llmisvc-controller-manager-586c85c465-kbml5.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-7d5cf78758-ghtbp.log ==================== {"level":"info","timestamp":"2026-07-06T15:56:14.217Z","caller":"cmd/main.go:65","message":"Loading database connection URL from secret..."} {"level":"info","timestamp":"2026-07-06T15:56:14.227Z","caller":"cmd/main.go:172","message":"Connecting to PostgreSQL database..."} {"level":"info","timestamp":"2026-07-06T15:56:14.227Z","caller":"cmd/main.go:101","message":"Metrics server starting","address":":9090"} {"level":"info","timestamp":"2026-07-06T15:56:14.252Z","caller":"api_keys/db_driver.go:90","message":"Database schema applied","version":3} {"level":"info","timestamp":"2026-07-06T15:56:14.252Z","caller":"api_keys/db_driver.go:57","message":"Connected to PostgreSQL database (schema applied)"} {"level":"info","timestamp":"2026-07-06T15:56:14.356Z","caller":"cmd/main.go:193","message":"Resolved gateway internal host for access probes","host":"maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local"} {"level":"info","timestamp":"2026-07-06T15:56:14.420Z","caller":"cmd/main.go:134","message":"Server starting","address":":8443","secure":true} [GIN] 2026/07/06 - 15:56:20 | 200 | 132.602µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:56:25 | 200 | 54.052µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:56:30 | 200 | 36.314µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:56:35 | 200 | 35.487µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:56:40 | 200 | 33.899µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:56:45 | 200 | 34.649µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:56:50 | 200 | 45.934µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:56:52 | 200 | 57.557µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:56:55 | 200 | 34.673µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:57:00 | 200 | 34.691µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:57:02 | 200 | 48.486µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:57:05 | 200 | 62.052µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:57:10 | 200 | 36.917µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:57:12 | 200 | 35.476µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:57:15 | 200 | 45.211µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:57:20 | 200 | 47.784µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:57:22 | 200 | 36.689µs | 10.129.2.2 | GET "/health" [GIN] 2026/07/06 - 15:57:25 | 200 | 37.215µs | 10.129.2.2 | GET "/health" ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-key-cleanup-29722515-hk9mw.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-key-cleanup-29722530-thnrf.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-api-key-cleanup-29722545-g47ng.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-controller-656b6867cb-9m2dm.log ==================== {"level":"info","ts":"2026-07-06T15:53:28Z","logger":"setup","msg":"subscription namespace already exists","namespace":"models-as-a-service","phase":"Active"} {"level":"info","ts":"2026-07-06T15:53:58Z","logger":"setup","msg":"subscription namespace already exists","namespace":"models-as-a-service","phase":"Active"} {"level":"info","ts":"2026-07-06T15:54:28Z","logger":"setup","msg":"subscription namespace already exists","namespace":"models-as-a-service","phase":"Active"} {"level":"info","ts":"2026-07-06T15:54:58Z","logger":"setup","msg":"subscription namespace already exists","namespace":"models-as-a-service","phase":"Active"} {"level":"info","ts":"2026-07-06T15:55:28Z","logger":"setup","msg":"subscription namespace already exists","namespace":"models-as-a-service","phase":"Active"} {"level":"info","ts":"2026-07-06T15:55:58Z","logger":"setup","msg":"subscription namespace already exists","namespace":"models-as-a-service","phase":"Active"} {"level":"info","ts":"2026-07-06T15:56:28Z","logger":"setup","msg":"subscription namespace already exists","namespace":"models-as-a-service","phase":"Active"} {"level":"info","ts":"2026-07-06T15:56:58Z","logger":"setup","msg":"subscription namespace already exists","namespace":"models-as-a-service","phase":"Active"} {"level":"info","ts":"2026-07-06T15:57:03Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"a8ca3790-8006-45be-b969-ac7b16ff5b7a","group":"perses.dev","kind":"PersesDatasource","name":"kuadrant-prometheus-datasource","namespace":"redhat-ods-applications"} {"level":"info","ts":"2026-07-06T15:57:03Z","msg":"skipping resource: optional CRD not yet registered, will apply once installed","controller":"tenant","controllerGroup":"maas.opendatahub.io","controllerKind":"Tenant","Tenant":{"name":"default-tenant","namespace":"models-as-a-service"},"namespace":"models-as-a-service","name":"default-tenant","reconcileID":"a8ca3790-8006-45be-b969-ac7b16ff5b7a","group":"perses.dev","kind":"PersesDashboard","name":"dashboard-3-maas-usage-admin","namespace":"redhat-ods-applications"} ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-postgres-d66f546cf-vsktb.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/mlflow-operator-controller-manager-8646f94cf4-r46v7.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-registry-operator-controller-manager-6cd8c54969-kkj6q.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-serving-api-78bbbb9bf8-5cnfr.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/notebook-controller-deployment-5d7d79cf5b-kvdwh.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-model-controller-6b647f87c-2chzc.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-notebook-controller-manager-7f6b58f576-gn4s9.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/postgres-595b8d5f6-smmgf.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/rhods-dashboard-6b5cc9785c-hcxj8.log ==================== {"level":30,"time":1783353205251,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h26","req":{"method":"GET","url":"/","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":47060},"msg":"incoming request"} {"level":30,"time":1783353205251,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h26","res":{"statusCode":200},"responseTime":0.3520517349243164,"msg":"request completed"} {"level":30,"time":1783353215852,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h27","req":{"method":"GET","url":"/api/health","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":38878},"msg":"incoming request"} {"level":30,"time":1783353215852,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h27","res":{"statusCode":200},"responseTime":0.26820898056030273,"msg":"request completed"} {"level":30,"time":1783353235251,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h28","req":{"method":"GET","url":"/","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":45248},"msg":"incoming request"} {"level":30,"time":1783353235252,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h28","res":{"statusCode":200},"responseTime":0.39231109619140625,"msg":"request completed"} {"level":30,"time":1783353245850,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h29","req":{"method":"GET","url":"/api/health","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":44212},"msg":"incoming request"} {"level":30,"time":1783353245850,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h29","res":{"statusCode":200},"responseTime":0.3859987258911133,"msg":"request completed"} {"level":30,"time":1783353265250,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2a","req":{"method":"GET","url":"/","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":48476},"msg":"incoming request"} {"level":30,"time":1783353265251,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2a","res":{"statusCode":200},"responseTime":0.5383720397949219,"msg":"request completed"} {"level":30,"time":1783353275850,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2b","req":{"method":"GET","url":"/api/health","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":36440},"msg":"incoming request"} {"level":30,"time":1783353275851,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2b","res":{"statusCode":200},"responseTime":0.2914156913757324,"msg":"request completed"} {"level":30,"time":1783353295251,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2c","req":{"method":"GET","url":"/","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":46382},"msg":"incoming request"} {"level":30,"time":1783353295252,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2c","res":{"statusCode":200},"responseTime":0.36113405227661133,"msg":"request completed"} {"level":30,"time":1783353305851,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2d","req":{"method":"GET","url":"/api/health","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":41242},"msg":"incoming request"} {"level":30,"time":1783353305851,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2d","res":{"statusCode":200},"responseTime":0.2564859390258789,"msg":"request completed"} {"level":30,"time":1783353325250,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2e","req":{"method":"GET","url":"/","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":50904},"msg":"incoming request"} {"level":30,"time":1783353325251,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2e","res":{"statusCode":200},"responseTime":0.3750319480895996,"msg":"request completed"} {"level":30,"time":1783353335851,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2f","req":{"method":"GET","url":"/api/health","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":44264},"msg":"incoming request"} {"level":30,"time":1783353335851,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2f","res":{"statusCode":200},"responseTime":0.3011918067932129,"msg":"request completed"} {"level":30,"time":1783353355253,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2g","req":{"method":"GET","url":"/","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":48392},"msg":"incoming request"} {"level":30,"time":1783353355254,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2g","res":{"statusCode":200},"responseTime":0.4789872169494629,"msg":"request completed"} {"level":30,"time":1783353365852,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2h","req":{"method":"GET","url":"/api/health","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":44210},"msg":"incoming request"} {"level":30,"time":1783353365853,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2h","res":{"statusCode":200},"responseTime":0.32643890380859375,"msg":"request completed"} {"level":30,"time":1783353385253,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2i","req":{"method":"GET","url":"/","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":36304},"msg":"incoming request"} {"level":30,"time":1783353385253,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2i","res":{"statusCode":200},"responseTime":0.37024879455566406,"msg":"request completed"} {"level":30,"time":1783353395853,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2j","req":{"method":"GET","url":"/api/health","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":45784},"msg":"incoming request"} {"level":30,"time":1783353395853,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2j","res":{"statusCode":200},"responseTime":0.32427406311035156,"msg":"request completed"} {"level":30,"time":1783353415253,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2k","req":{"method":"GET","url":"/","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":48726},"msg":"incoming request"} {"level":30,"time":1783353415253,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2k","res":{"statusCode":200},"responseTime":0.3364138603210449,"msg":"request completed"} {"level":30,"time":1783353425852,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2l","req":{"method":"GET","url":"/api/health","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":52954},"msg":"incoming request"} {"level":30,"time":1783353425852,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2l","res":{"statusCode":200},"responseTime":0.25837182998657227,"msg":"request completed"} {"level":30,"time":1783353445252,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2m","req":{"method":"GET","url":"/","hostname":"127.0.0.1:8080","remoteAddress":"127.0.0.1","remotePort":34104},"msg":"incoming request"} {"level":30,"time":1783353445252,"pid":12,"hostname":"rhods-dashboard-6b5cc9785c-hcxj8","reqId":"req-h2m","res":{"statusCode":200},"responseTime":0.45395421981811523,"msg":"request completed"} time=2026-07-06T15:53:30.126Z level=WARN msg="MLflow CR auto-discovery failed, MLflow endpoints will return 503" error="no MLflow CR found in the cluster" time=2026-07-06T15:54:00.130Z level=WARN msg="MLflow CR auto-discovery failed, MLflow endpoints will return 503" error="no MLflow CR found in the cluster" time=2026-07-06T15:54:30.132Z level=WARN msg="MLflow CR auto-discovery failed, MLflow endpoints will return 503" error="no MLflow CR found in the cluster" time=2026-07-06T15:55:00.130Z level=WARN msg="MLflow CR auto-discovery failed, MLflow endpoints will return 503" error="no MLflow CR found in the cluster" time=2026-07-06T15:55:30.133Z level=WARN msg="MLflow CR auto-discovery failed, MLflow endpoints will return 503" error="no MLflow CR found in the cluster" time=2026-07-06T15:56:00.132Z level=WARN msg="MLflow CR auto-discovery failed, MLflow endpoints will return 503" error="no MLflow CR found in the cluster" time=2026-07-06T15:56:30.132Z level=WARN msg="MLflow CR auto-discovery failed, MLflow endpoints will return 503" error="no MLflow CR found in the cluster" time=2026-07-06T15:57:00.140Z level=WARN msg="MLflow CR auto-discovery failed, MLflow endpoints will return 503" error="no MLflow CR found in the cluster" ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-controller-57c5867998-2vtgh.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-webhook-56c5bb987f-mv7t6.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/trustyai-service-operator-controller-manager-6f7f88c874-lx676.log ==================== W0706 15:53:55.185557 1 warnings.go:70] This version is deprecated. Use v1beta2 instead. ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-7cc8755dc9-p8vfh.log ==================== ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-postgres-686d85b578-8r6r4.log ====================