==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== 73m Warning FailedScheduling pod/rhods-dashboard-558f548bfd-2767z 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 74m Warning FailedScheduling pod/rhods-dashboard-558f548bfd-2767z 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 74m Warning FailedScheduling pod/spark-pi-d921429f7e3d5979-exec-1 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 100m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-7bf7988d75s2 MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 100m Warning FailedMount pod/odh-model-controller-65fdb9c5c-lhnjg MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 100m Warning FailedMount pod/model-registry-operator-controller-manager-5699f87db4-d5spn MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 100m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 100m Warning FailedMount pod/odh-notebook-controller-manager-945b948d4-q9ddr MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 100m Warning FailedCreate replicaset/notebook-controller-deployment-647b7d4d45 Error creating: pods "notebook-controller-deployment-647b7d4d45-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 100m Warning FailedMount pod/kuberay-operator-6586cdc875-lgw4n MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 100m Warning FailedCreate replicaset/rhods-dashboard-558f548bfd Error creating: pods "rhods-dashboard-558f548bfd-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 99m Warning FailedMount pod/rhods-dashboard-558f548bfd-w6qjd MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 99m Warning FailedMount pod/rhods-dashboard-558f548bfd-dlnf6 MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 99m Warning FailedMount pod/kserve-controller-manager-dd9f6cf9-jq2f4 MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 99m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-5dccbd95db Error creating: pods "trustyai-service-operator-controller-manager-5dccbd95db-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 99m Warning FailedMount pod/trustyai-service-operator-controller-manager-5dccbd95db-lwgtr MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 76m Warning FailedCreate replicaset/spark-operator-webhook-5f8768dbfd Error creating: pods "spark-operator-webhook-5f8768dbfd-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-webhook: serviceaccount "spark-operator-webhook" not found 76m Warning FailedMount pod/ogx-k8s-operator-controller-manager-6f9798dc69-wsbj4 MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 75m Warning FailedCreate replicaset/spark-operator-controller-5dd7d48f87 Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 75m Warning FailedMount pod/odh-model-controller-65fdb9c5c-z8mtx MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 75m Warning FailedMount pod/odh-notebook-controller-manager-945b948d4-hv4xk MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 75m Warning FailedMount pod/model-registry-operator-controller-manager-5699f87db4-zxdnk MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 75m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 75m Warning FailedMount pod/kserve-controller-manager-dd9f6cf9-szwt6 MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 75m Warning FailedMount pod/llmisvc-controller-manager-7b46b9fbbb-8ns66 MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 75m Warning FailedMount pod/kuberay-operator-6586cdc875-5f8mb MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 75m Warning CalculateExpectedPodCountFailed poddisruptionbudget/ogx-k8s-operator-controller-manager-pdb Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-6f9798dc69-wsbj4" 75m Warning CalculateExpectedPodCountFailed poddisruptionbudget/ogx-k8s-operator-controller-manager-pdb Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-6f9798dc69-nt9mg" 74m Warning Unhealthy pod/kubeflow-trainer-controller-manager-b5fd84b9c-5fn9z Readiness probe failed: Get "http://10.134.0.65:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 74m Warning FailedMount pod/spark-pi-d9es3ejkisas7b6mh690-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-e116a09f7e3cf3d7-conf-map" not found 74m Warning FailedMount pod/spark-pi-d9es3ejkisas7b6mh690-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d9es3ejkisas7b6mh690-375e4d9f7e3cdec8-driver-podspec-conf-map" not found 74m Warning FailedCreate replicaset/rhods-dashboard-558f548bfd Error creating: pods "rhods-dashboard-558f548bfd-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 72m Warning FailedMount pod/scheduled-spark-pi-d9es42jkisas7b6mh69g-1784529448000882054-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d9es42jkisas7b6mh69g-1784529448000882054-02c8329f7e3e88ee-driver-podspec-conf-map" not found 72m Warning FailedMount pod/scheduled-spark-pi-d9es42jkisas7b6mh69g-1784529448000882054-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-1f3ede9f7e3e9a2b-conf-map" not found 70m Warning FailedMount pod/model-registry-operator-controller-manager-5699f87db4-rjdcn MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 70m Warning FailedMount pod/odh-model-controller-65fdb9c5c-rcxvr MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 70m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 70m Warning FailedMount pod/model-serving-api-5679d6b4f-s8sdl MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 70m Warning FailedMount pod/llmisvc-controller-manager-7b46b9fbbb-x686m MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 70m Warning FailedCreate replicaset/workload-variant-autoscaler-controller-manager-7f5b9cd77b Error creating: pods "workload-variant-autoscaler-controller-manager-7f5b9cd77b-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 70m Warning FailedMount pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-22zmx MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 67m Warning FailedMount pod/odh-model-controller-bd4c5cbcb-zqh2z MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 67m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 67m Warning FailedMount pod/mlflow-operator-controller-manager-7c46b854f7-kxs7r MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 66m Warning FailedMount pod/llmisvc-controller-manager-7b46b9fbbb-2tz2p MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 65m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-5dccbd95db Error creating: pods "trustyai-service-operator-controller-manager-5dccbd95db-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 65m Warning FailedMount pod/trustyai-service-operator-controller-manager-5dccbd95db-nz4vn MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 63m Warning FailedToUpdateEndpoint endpoints/model-serving-api Failed to update endpoint redhat-ods-applications/model-serving-api: Operation cannot be fulfilled on endpoints "model-serving-api": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/model-serving-api, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c8480568-5cbb-47b1-9b97-caf4f9b57560, UID in object meta: 62m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 62m Warning FailedMount pod/odh-model-controller-bd4c5cbcb-jggsq MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 62m Warning FailedMount pod/model-serving-api-5679d6b4f-5kxpv MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 61m Warning FailedCreate replicaset/maas-api-5b7784b768 Error creating: pods "maas-api-5b7784b768-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 60m Warning FailedMount pod/llmisvc-controller-manager-7b46b9fbbb-22ngc MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 60m Warning FailedMount pod/kserve-controller-manager-dd9f6cf9-8nnzz MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 60m Warning FailedMount pod/odh-model-controller-bd4c5cbcb-84rz6 MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 60m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 59m Warning FailedMount pod/llmisvc-controller-manager-7b46b9fbbb-hb525 MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 59m Warning FailedMount pod/odh-model-controller-bd4c5cbcb-q7c6k MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 59m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 58m Warning FailedMount pod/llmisvc-controller-manager-7b46b9fbbb-94kgg MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 57m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-7bf7988r7lkx MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 57m Warning FailedMount pod/model-registry-operator-controller-manager-5699f87db4-wqkb9 MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 57m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-5dccbd95db Error creating: pods "trustyai-service-operator-controller-manager-5dccbd95db-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 57m Warning FailedMount pod/odh-notebook-controller-manager-945b948d4-q82dg MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 57m Warning FailedCreate replicaset/notebook-controller-deployment-647b7d4d45 Error creating: pods "notebook-controller-deployment-647b7d4d45-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 57m Warning FailedCreate replicaset/spark-operator-controller-5dd7d48f87 Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 57m Warning FailedMount pod/trustyai-service-operator-controller-manager-5dccbd95db-4kgms MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 57m Warning FailedCreate replicaset/rhods-dashboard-558f548bfd Error creating: pods "rhods-dashboard-558f548bfd-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 57m Warning FailedMount pod/kuberay-operator-6586cdc875-zx2z9 MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 9m Warning FailedCreate replicaset/spark-operator-controller-5dd7d48f87 Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 7m50s Warning FailedMount pod/spark-pi-d9et2r3kisas7bb829i0-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-f635179f7e7a4012-conf-map" not found 7m50s Warning FailedMount pod/spark-pi-d9et2r3kisas7bb829i0-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d9et2r3kisas7bb829i0-95c50f9f7e7a2d54-driver-podspec-conf-map" not found 6m31s Warning FailedMount pod/scheduled-spark-pi-d9et37jkisas7bb829ig-1784533436001181118-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-0d12169f7e7b743b-conf-map" not found 6m21s Warning FailedMount pod/spark-pi-438b6a9f7e7b95f7-exec-1 MountVolume.SetUp failed for volume "spark-conf-volume-exec" : configmap "spark-exec-c5c8e89f7e7b9fae-conf-map" not found 3m49s Warning FailedCreate replicaset/spark-operator-controller-5dd7d48f87 Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 100m Warning AppliedWithWarnings installplan/install-mm84l 1 warning(s) generated during installation of operator "rhods-operator.3.5.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 37m Warning Deleting modelregistry/postgres-registry-298031 Custom Resource postgres-registry-298031 is being deleted from the namespace rhoai-model-registries 30m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 37m Warning Deleting modelregistry/postgres-registry-298031 Custom Resource postgres-registry-298031 is being deleted from the namespace rhoai-model-registries 30m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 37m Warning Deleting modelregistry/e2e-test-registry-298031 Custom Resource e2e-test-registry-298031 is being deleted from the namespace rhoai-model-registries 36m Warning Deleting modelregistry/default-db-registry-298031 Custom Resource default-db-registry-298031 is being deleted from the namespace rhoai-model-registries 36m Warning Deleting modelregistry/default-db-registry-298031 Custom Resource default-db-registry-298031 is being deleted from the namespace rhoai-model-registries 100m Warning FailedMount pod/model-catalog-7c7d9467f5-qvsjq MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 99m Warning Unhealthy pod/model-catalog-7c7d9467f5-qvsjq Readiness probe failed: Get "http://10.134.0.54:8080/readyz": dial tcp 10.134.0.54:8080: connect: connection refused 75m Warning FailedMount pod/model-catalog-7c7d9467f5-sxvd8 MountVolume.SetUp failed for volume "kube-rbac-proxy-tls" : secret "model-catalog-kube-rbac-proxy" not found 75m Warning FailedMount pod/model-catalog-7c7d9467f5-sxvd8 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 74m Warning Unhealthy pod/model-catalog-7c7d9467f5-sxvd8 Readiness probe failed: Get "http://10.134.0.75:8080/readyz": dial tcp 10.134.0.75:8080: connect: connection refused 74m Warning FailedToUpdateEndpointSlices service/model-catalog Error updating Endpoint Slices for Service rhoai-model-registries/model-catalog: failed to update model-catalog-hh5lq EndpointSlice for Service rhoai-model-registries/model-catalog: endpointslices.discovery.k8s.io "model-catalog-hh5lq" not found 70m Warning FailedMount pod/model-catalog-7c7d9467f5-scprg MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 70m Warning Unhealthy pod/model-catalog-7c7d9467f5-scprg Readiness probe failed: Get "http://10.132.0.60:8080/readyz": dial tcp 10.132.0.60:8080: connect: connection refused 67m Warning FailedToUpdateEndpointSlices service/model-catalog-postgres Error updating Endpoint Slices for Service rhoai-model-registries/model-catalog-postgres: failed to update model-catalog-postgres-jkg4z EndpointSlice for Service rhoai-model-registries/model-catalog-postgres: endpointslices.discovery.k8s.io "model-catalog-postgres-jkg4z" not found 57m Warning FailedMount pod/model-catalog-7c7d9467f5-7sbd8 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 57m Warning Unhealthy pod/model-catalog-7c7d9467f5-7sbd8 Readiness probe failed: Get "http://10.134.0.107:8080/readyz": dial tcp 10.134.0.107:8080: connect: connection refused 39m Warning Unhealthy pod/mysql-db-registry-298031-8cd86c49d-lwkrp Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 38m Warning FailedMount pod/e2e-test-registry-298031-66576bf868-phs6p MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "e2e-test-registry-298031-kube-rbac-proxy" not found 38m Warning FailedMount pod/e2e-test-registry-298031-66576bf868-phs6p MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "e2e-test-registry-298031-kube-rbac-proxy" not found 37m Warning FailedMount pod/default-db-registry-298031-6575d56cf5-h9v6s MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "default-db-registry-298031-kube-rbac-proxy" not found 37m Warning FailedMount pod/default-db-registry-298031-6575d56cf5-h9v6s MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "default-db-registry-298031-kube-rbac-proxy" not found 36m Warning Unhealthy pod/default-db-registry-298031-6575d56cf5-h9v6s Startup probe failed: Get "http://10.134.0.120:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 36m Warning Unhealthy pod/default-db-registry-298031-6575d56cf5-h9v6s Startup probe failed: HTTP probe failed with statuscode: 503 36m Warning Unhealthy pod/mysql-db-registry-298031-8cd86c49d-lwkrp Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 35m Warning Unhealthy pod/default-db-registry-298031-postgres-5448fc9c6-4mkqx Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 35m Warning Unhealthy pod/model-registry-db-662874-864c6b9cc8-lfmd6 Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 35m Warning FailedMount pod/test-model-registry-662874-76fcb4b8d6-x7sxb MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "test-model-registry-662874-kube-rbac-proxy" not found 35m Warning FailedMount pod/test-model-registry-662874-76fcb4b8d6-x7sxb MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "test-model-registry-662874-kube-rbac-proxy" not found --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-07-20T07:50:37Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.21.25 Kubernetes Version: v1.34.9 --- Namespaces --- namespace/opendatahub namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.1 Authorino Operator 1.4.1 authorino-operator.v1.3.2 Succeeded cert-manager-operator.v1.20.0 cert-manager Operator for Red Hat OpenShift 1.20.0 cert-manager-operator.v1.19.0 Succeeded custom-metrics-autoscaler.v2.19.0-2 Custom Metrics Autoscaler 2.19.0-2 custom-metrics-autoscaler.v2.19.0-1 Succeeded dns-operator.v1.4.0 DNS Operator 1.4.0 dns-operator.v1.3.1 Succeeded kueue-operator.v1.3.1 Red Hat build of Kueue 1.3.1 Succeeded limitador-operator.v1.4.0 Limitador Operator 1.4.0 limitador-operator.v1.3.1 Succeeded rhcl-operator.v1.4.1 Red Hat Connectivity Link 1.4.1 rhcl-operator.v1.3.5 Succeeded rhods-operator.3.5.0-ea.2 Red Hat OpenShift AI 3.5.0-ea.2 Succeeded --- CSVs not in Succeeded phase --- (all CSVs report phase Succeeded) --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aigateway: managementState=n/a aipipelines: managementState=Managed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Removed llamastackoperator: managementState=n/a mlflowoperator: managementState=Removed modelregistry: managementState=Managed ogx: managementState=Removed ray: managementState=Managed sparkoperator: managementState=Managed trainer: managementState=Removed trainingoperator: managementState=Removed trustyai: managementState=Managed workbenches: managementState=Managed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= AIPipelinesReady: status=True reason= message= DashboardReady: status=True reason= message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=False reason=Removed message=Component ManagementState is set to Removed LLM-D-WVADependencies: status=True reason= message= MLflowOperatorReady: status=False reason=Removed message=Component ManagementState is set to Removed ModelControllerReady: status=True reason= message= ModelRegistryReady: status=True reason= message= ModelsAsServiceReady: status=False reason=Removed message=Component ManagementState is set to Removed OGXReady: status=False reason=Removed message=Component ManagementState is set to Removed RayReady: status=True reason= message= SparkOperatorReady: status=True reason= message= TrainerReady: status=False reason=Removed message=Component ManagementState is set to Removed TrainingOperatorReady: status=False reason=Removed message=Component ManagementState is set to Removed TrustyAIReady: status=True reason= message= WorkbenchesReady: status=True reason= message= --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 100m Ready 2026-07-20T06:10:09Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in DSCI CR OpenTelemetryCollectorAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Metrics not configured in DSCI CR Traces not configured in DSCI CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in DSCI CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR PersesAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR MonitoringReady: status=True reason=Ready message=Monitoring stack is initialized --- Component CRs (Ready) --- NAME READY REASON URL dashboard.components.platform.opendatahub.io/default-dashboard True NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON URI modelcontroller.components.platform.opendatahub.io/default-modelcontroller True NAME READY REASON modelregistry.components.platform.opendatahub.io/default-modelregistry True NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON sparkoperator.components.platform.opendatahub.io/default-sparkoperator True NAME READY REASON trustyai.components.platform.opendatahub.io/default-trustyai True NAME READY REASON workbenches.components.platform.opendatahub.io/default-workbenches True --- Deployments not fully ready (ODH/RHOAI namespaces) --- (all deployments report ready replicas == desired) --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- (all pods Running or Succeeded) --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/opendatahub --- (no events) --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 73m Warning FailedScheduling pod/rhods-dashboard-558f548bfd-2767z 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 74m Warning FailedScheduling pod/rhods-dashboard-558f548bfd-2767z 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 74m Warning FailedScheduling pod/spark-pi-d921429f7e3d5979-exec-1 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 100m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-7bf7988d75s2 MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 100m Warning FailedMount pod/odh-model-controller-65fdb9c5c-lhnjg MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 100m Warning FailedMount pod/model-registry-operator-controller-manager-5699f87db4-d5spn MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 100m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 100m Warning FailedMount pod/odh-notebook-controller-manager-945b948d4-q9ddr MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 100m Warning FailedCreate replicaset/notebook-controller-deployment-647b7d4d45 Error creating: pods "notebook-controller-deployment-647b7d4d45-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 100m Warning FailedMount pod/kuberay-operator-6586cdc875-lgw4n MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 100m Warning FailedCreate replicaset/rhods-dashboard-558f548bfd Error creating: pods "rhods-dashboard-558f548bfd-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 99m Warning FailedMount pod/rhods-dashboard-558f548bfd-w6qjd MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 99m Warning FailedMount pod/rhods-dashboard-558f548bfd-dlnf6 MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 99m Warning FailedMount pod/kserve-controller-manager-dd9f6cf9-jq2f4 MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 99m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-5dccbd95db Error creating: pods "trustyai-service-operator-controller-manager-5dccbd95db-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 99m Warning FailedMount pod/trustyai-service-operator-controller-manager-5dccbd95db-lwgtr MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 76m Warning FailedCreate replicaset/spark-operator-webhook-5f8768dbfd Error creating: pods "spark-operator-webhook-5f8768dbfd-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-webhook: serviceaccount "spark-operator-webhook" not found 76m Warning FailedMount pod/ogx-k8s-operator-controller-manager-6f9798dc69-wsbj4 MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 75m Warning FailedCreate replicaset/spark-operator-controller-5dd7d48f87 Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 75m Warning FailedMount pod/odh-model-controller-65fdb9c5c-z8mtx MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 75m Warning FailedMount pod/odh-notebook-controller-manager-945b948d4-hv4xk MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 75m Warning FailedMount pod/model-registry-operator-controller-manager-5699f87db4-zxdnk MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 75m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 75m Warning FailedMount pod/kserve-controller-manager-dd9f6cf9-szwt6 MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 75m Warning FailedMount pod/llmisvc-controller-manager-7b46b9fbbb-8ns66 MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 75m Warning FailedMount pod/kuberay-operator-6586cdc875-5f8mb MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 75m Warning CalculateExpectedPodCountFailed poddisruptionbudget/ogx-k8s-operator-controller-manager-pdb Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-6f9798dc69-wsbj4" 75m Warning CalculateExpectedPodCountFailed poddisruptionbudget/ogx-k8s-operator-controller-manager-pdb Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-6f9798dc69-nt9mg" 74m Warning Unhealthy pod/kubeflow-trainer-controller-manager-b5fd84b9c-5fn9z Readiness probe failed: Get "http://10.134.0.65:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 74m Warning FailedMount pod/spark-pi-d9es3ejkisas7b6mh690-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-e116a09f7e3cf3d7-conf-map" not found 74m Warning FailedMount pod/spark-pi-d9es3ejkisas7b6mh690-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d9es3ejkisas7b6mh690-375e4d9f7e3cdec8-driver-podspec-conf-map" not found 74m Warning FailedCreate replicaset/rhods-dashboard-558f548bfd Error creating: pods "rhods-dashboard-558f548bfd-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 72m Warning FailedMount pod/scheduled-spark-pi-d9es42jkisas7b6mh69g-1784529448000882054-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d9es42jkisas7b6mh69g-1784529448000882054-02c8329f7e3e88ee-driver-podspec-conf-map" not found 72m Warning FailedMount pod/scheduled-spark-pi-d9es42jkisas7b6mh69g-1784529448000882054-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-1f3ede9f7e3e9a2b-conf-map" not found 70m Warning FailedMount pod/model-registry-operator-controller-manager-5699f87db4-rjdcn MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 70m Warning FailedMount pod/odh-model-controller-65fdb9c5c-rcxvr MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 70m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 70m Warning FailedMount pod/model-serving-api-5679d6b4f-s8sdl MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 70m Warning FailedMount pod/llmisvc-controller-manager-7b46b9fbbb-x686m MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 70m Warning FailedCreate replicaset/workload-variant-autoscaler-controller-manager-7f5b9cd77b Error creating: pods "workload-variant-autoscaler-controller-manager-7f5b9cd77b-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 70m Warning FailedMount pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-22zmx MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 67m Warning FailedMount pod/odh-model-controller-bd4c5cbcb-zqh2z MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 67m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 67m Warning FailedMount pod/mlflow-operator-controller-manager-7c46b854f7-kxs7r MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 66m Warning FailedMount pod/llmisvc-controller-manager-7b46b9fbbb-2tz2p MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 65m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-5dccbd95db Error creating: pods "trustyai-service-operator-controller-manager-5dccbd95db-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 65m Warning FailedMount pod/trustyai-service-operator-controller-manager-5dccbd95db-nz4vn MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 63m Warning FailedToUpdateEndpoint endpoints/model-serving-api Failed to update endpoint redhat-ods-applications/model-serving-api: Operation cannot be fulfilled on endpoints "model-serving-api": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/model-serving-api, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c8480568-5cbb-47b1-9b97-caf4f9b57560, UID in object meta: 62m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 62m Warning FailedMount pod/odh-model-controller-bd4c5cbcb-jggsq MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 62m Warning FailedMount pod/model-serving-api-5679d6b4f-5kxpv MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 61m Warning FailedCreate replicaset/maas-api-5b7784b768 Error creating: pods "maas-api-5b7784b768-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 60m Warning FailedMount pod/llmisvc-controller-manager-7b46b9fbbb-22ngc MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 60m Warning FailedMount pod/kserve-controller-manager-dd9f6cf9-8nnzz MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 60m Warning FailedMount pod/odh-model-controller-bd4c5cbcb-84rz6 MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 60m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 59m Warning FailedMount pod/llmisvc-controller-manager-7b46b9fbbb-hb525 MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 59m Warning FailedMount pod/odh-model-controller-bd4c5cbcb-q7c6k MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 59m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 58m Warning FailedMount pod/llmisvc-controller-manager-7b46b9fbbb-94kgg MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 57m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-7bf7988r7lkx MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 57m Warning FailedMount pod/model-registry-operator-controller-manager-5699f87db4-wqkb9 MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 57m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-5dccbd95db Error creating: pods "trustyai-service-operator-controller-manager-5dccbd95db-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 57m Warning FailedMount pod/odh-notebook-controller-manager-945b948d4-q82dg MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 57m Warning FailedCreate replicaset/notebook-controller-deployment-647b7d4d45 Error creating: pods "notebook-controller-deployment-647b7d4d45-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 57m Warning FailedCreate replicaset/spark-operator-controller-5dd7d48f87 Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 57m Warning FailedMount pod/trustyai-service-operator-controller-manager-5dccbd95db-4kgms MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 57m Warning FailedCreate replicaset/rhods-dashboard-558f548bfd Error creating: pods "rhods-dashboard-558f548bfd-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 57m Warning FailedMount pod/kuberay-operator-6586cdc875-zx2z9 MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 9m Warning FailedCreate replicaset/spark-operator-controller-5dd7d48f87 Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 7m50s Warning FailedMount pod/spark-pi-d9et2r3kisas7bb829i0-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-f635179f7e7a4012-conf-map" not found 7m50s Warning FailedMount pod/spark-pi-d9et2r3kisas7bb829i0-driver MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d9et2r3kisas7bb829i0-95c50f9f7e7a2d54-driver-podspec-conf-map" not found 6m31s Warning FailedMount pod/scheduled-spark-pi-d9et37jkisas7bb829ig-1784533436001181118-driver MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-0d12169f7e7b743b-conf-map" not found 6m21s Warning FailedMount pod/spark-pi-438b6a9f7e7b95f7-exec-1 MountVolume.SetUp failed for volume "spark-conf-volume-exec" : configmap "spark-exec-c5c8e89f7e7b9fae-conf-map" not found 3m49s Warning FailedCreate replicaset/spark-operator-controller-5dd7d48f87 Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- LAST SEEN TYPE REASON OBJECT MESSAGE 100m Warning AppliedWithWarnings installplan/install-mm84l 1 warning(s) generated during installation of operator "rhods-operator.3.5.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 --- namespace/rhoai-model-registries --- LAST SEEN TYPE REASON OBJECT MESSAGE 37m Warning Deleting modelregistry/postgres-registry-298031 Custom Resource postgres-registry-298031 is being deleted from the namespace rhoai-model-registries 30m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 37m Warning Deleting modelregistry/postgres-registry-298031 Custom Resource postgres-registry-298031 is being deleted from the namespace rhoai-model-registries 30m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 37m Warning Deleting modelregistry/e2e-test-registry-298031 Custom Resource e2e-test-registry-298031 is being deleted from the namespace rhoai-model-registries 36m Warning Deleting modelregistry/default-db-registry-298031 Custom Resource default-db-registry-298031 is being deleted from the namespace rhoai-model-registries 36m Warning Deleting modelregistry/default-db-registry-298031 Custom Resource default-db-registry-298031 is being deleted from the namespace rhoai-model-registries 100m Warning FailedMount pod/model-catalog-7c7d9467f5-qvsjq MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 99m Warning Unhealthy pod/model-catalog-7c7d9467f5-qvsjq Readiness probe failed: Get "http://10.134.0.54:8080/readyz": dial tcp 10.134.0.54:8080: connect: connection refused 75m Warning FailedMount pod/model-catalog-7c7d9467f5-sxvd8 MountVolume.SetUp failed for volume "kube-rbac-proxy-tls" : secret "model-catalog-kube-rbac-proxy" not found 75m Warning FailedMount pod/model-catalog-7c7d9467f5-sxvd8 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 74m Warning Unhealthy pod/model-catalog-7c7d9467f5-sxvd8 Readiness probe failed: Get "http://10.134.0.75:8080/readyz": dial tcp 10.134.0.75:8080: connect: connection refused 74m Warning FailedToUpdateEndpointSlices service/model-catalog Error updating Endpoint Slices for Service rhoai-model-registries/model-catalog: failed to update model-catalog-hh5lq EndpointSlice for Service rhoai-model-registries/model-catalog: endpointslices.discovery.k8s.io "model-catalog-hh5lq" not found 70m Warning FailedMount pod/model-catalog-7c7d9467f5-scprg MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 70m Warning Unhealthy pod/model-catalog-7c7d9467f5-scprg Readiness probe failed: Get "http://10.132.0.60:8080/readyz": dial tcp 10.132.0.60:8080: connect: connection refused 67m Warning FailedToUpdateEndpointSlices service/model-catalog-postgres Error updating Endpoint Slices for Service rhoai-model-registries/model-catalog-postgres: failed to update model-catalog-postgres-jkg4z EndpointSlice for Service rhoai-model-registries/model-catalog-postgres: endpointslices.discovery.k8s.io "model-catalog-postgres-jkg4z" not found 57m Warning FailedMount pod/model-catalog-7c7d9467f5-7sbd8 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 57m Warning Unhealthy pod/model-catalog-7c7d9467f5-7sbd8 Readiness probe failed: Get "http://10.134.0.107:8080/readyz": dial tcp 10.134.0.107:8080: connect: connection refused 39m Warning Unhealthy pod/mysql-db-registry-298031-8cd86c49d-lwkrp Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 38m Warning FailedMount pod/e2e-test-registry-298031-66576bf868-phs6p MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "e2e-test-registry-298031-kube-rbac-proxy" not found 38m Warning FailedMount pod/e2e-test-registry-298031-66576bf868-phs6p MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "e2e-test-registry-298031-kube-rbac-proxy" not found 37m Warning FailedMount pod/default-db-registry-298031-6575d56cf5-h9v6s MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "default-db-registry-298031-kube-rbac-proxy" not found 37m Warning FailedMount pod/default-db-registry-298031-6575d56cf5-h9v6s MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "default-db-registry-298031-kube-rbac-proxy" not found 36m Warning Unhealthy pod/default-db-registry-298031-6575d56cf5-h9v6s Startup probe failed: Get "http://10.134.0.120:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 36m Warning Unhealthy pod/default-db-registry-298031-6575d56cf5-h9v6s Startup probe failed: HTTP probe failed with statuscode: 503 36m Warning Unhealthy pod/mysql-db-registry-298031-8cd86c49d-lwkrp Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 35m Warning Unhealthy pod/default-db-registry-298031-postgres-5448fc9c6-4mkqx Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 35m Warning Unhealthy pod/model-registry-db-662874-864c6b9cc8-lfmd6 Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 35m Warning FailedMount pod/test-model-registry-662874-76fcb4b8d6-x7sxb MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "test-model-registry-662874-kube-rbac-proxy" not found 35m Warning FailedMount pod/test-model-registry-662874-76fcb4b8d6-x7sxb MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "test-model-registry-662874-kube-rbac-proxy" not found --- namespace/rhods-notebooks --- LAST SEEN TYPE REASON OBJECT MESSAGE --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-07-20T05:45:40Z) --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-75864bf978-5mqpp.log ==================== ERROR: cannot export logs for pod redhat-ods-operator/rhods-operator-75864bf978-5mqpp: Error from server: Get "https://10.0.132.133:10250/containerLogs/redhat-ods-operator/rhods-operator-75864bf978-5mqpp/copy-manifests?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== OPERATOR POD LOG: rhods-operator-75864bf978-9tbmn.log ==================== ERROR: cannot export logs for pod redhat-ods-operator/rhods-operator-75864bf978-9tbmn: Error from server: Get "https://10.0.133.135:10250/containerLogs/redhat-ods-operator/rhods-operator-75864bf978-9tbmn/copy-manifests?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== OPERATOR POD LOG: rhods-operator-75864bf978-v2ljz.log ==================== ERROR: cannot export logs for pod redhat-ods-operator/rhods-operator-75864bf978-v2ljz: Error from server: Get "https://10.0.138.210:10250/containerLogs/redhat-ods-operator/rhods-operator-75864bf978-v2ljz/copy-manifests?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-85d4f96fcf-6fd62.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/dashboard-redirect-85d4f96fcf-6fd62: Error from server: Get "https://10.0.133.135:10250/containerLogs/redhat-ods-applications/dashboard-redirect-85d4f96fcf-6fd62/nginx?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/data-science-pipelines-operator-controller-manager-7bf7988r7lkx.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/data-science-pipelines-operator-controller-manager-7bf7988r7lkx: Error from server: Get "https://10.0.133.135:10250/containerLogs/redhat-ods-applications/data-science-pipelines-operator-controller-manager-7bf7988r7lkx/manager?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/feast-operator-controller-manager-7b7cd769f7-4z25g.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/feast-operator-controller-manager-7b7cd769f7-4z25g: Error from server: Get "https://10.0.133.135:10250/containerLogs/redhat-ods-applications/feast-operator-controller-manager-7b7cd769f7-4z25g/manager?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-controller-manager-dd9f6cf9-6p95x.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/kserve-controller-manager-dd9f6cf9-6p95x: Error from server: Get "https://10.0.133.135:10250/containerLogs/redhat-ods-applications/kserve-controller-manager-dd9f6cf9-6p95x/manager?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/kuberay-operator-6586cdc875-zx2z9.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/kuberay-operator-6586cdc875-zx2z9: Error from server: Get "https://10.0.138.210:10250/containerLogs/redhat-ods-applications/kuberay-operator-6586cdc875-zx2z9/kuberay-operator?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-94kgg.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-94kgg: Error from server: Get "https://10.0.138.210:10250/containerLogs/redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-94kgg/manager?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-postgres-bb9d55498-x8bsl.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/maas-postgres-bb9d55498-x8bsl: Error from server: Get "https://10.0.133.135:10250/containerLogs/redhat-ods-applications/maas-postgres-bb9d55498-x8bsl/postgres?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/model-registry-operator-controller-manager-68878656b9-bwkhn.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/model-registry-operator-controller-manager-68878656b9-bwkhn: Error from server: Get "https://10.0.138.210:10250/containerLogs/redhat-ods-applications/model-registry-operator-controller-manager-68878656b9-bwkhn/manager?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/model-serving-api-5679d6b4f-xvz8f.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/model-serving-api-5679d6b4f-xvz8f: Error from server: Get "https://10.0.133.135:10250/containerLogs/redhat-ods-applications/model-serving-api-5679d6b4f-xvz8f/server?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/notebook-controller-deployment-647b7d4d45-f7mxz.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/notebook-controller-deployment-647b7d4d45-f7mxz: Error from server: Get "https://10.0.132.133:10250/containerLogs/redhat-ods-applications/notebook-controller-deployment-647b7d4d45-f7mxz/manager?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-model-controller-65fdb9c5c-2jc5m.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/odh-model-controller-65fdb9c5c-2jc5m: Error from server: Get "https://10.0.132.133:10250/containerLogs/redhat-ods-applications/odh-model-controller-65fdb9c5c-2jc5m/manager?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-notebook-controller-manager-945b948d4-q82dg.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/odh-notebook-controller-manager-945b948d4-q82dg: Error from server: Get "https://10.0.138.210:10250/containerLogs/redhat-ods-applications/odh-notebook-controller-manager-945b948d4-q82dg/manager?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/rhods-dashboard-558f548bfd-kzkgs.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/rhods-dashboard-558f548bfd-kzkgs: Error from server: Get "https://10.0.133.135:10250/containerLogs/redhat-ods-applications/rhods-dashboard-558f548bfd-kzkgs/rhods-dashboard?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-controller-5dd7d48f87-jbp8t.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-jbp8t: Error from server: Get "https://10.0.138.210:10250/containerLogs/redhat-ods-applications/spark-operator-controller-5dd7d48f87-jbp8t/controller?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-webhook-5f8768dbfd-t2tdd.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-t2tdd: Error from server: Get "https://10.0.138.210:10250/containerLogs/redhat-ods-applications/spark-operator-webhook-5f8768dbfd-t2tdd/webhook?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: redhat-ods-applications/trustyai-service-operator-controller-manager-5dccbd95db-4kgms.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/trustyai-service-operator-controller-manager-5dccbd95db-4kgms: Error from server: Get "https://10.0.133.135:10250/containerLogs/redhat-ods-applications/trustyai-service-operator-controller-manager-5dccbd95db-4kgms/manager?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-7c7d9467f5-7sbd8.log ==================== ERROR: cannot export logs for pod rhoai-model-registries/model-catalog-7c7d9467f5-7sbd8: Error from server: Get "https://10.0.138.210:10250/containerLogs/rhoai-model-registries/model-catalog-7c7d9467f5-7sbd8/catalog-data-init?sinceTime=2026-07-20T05%3A45%3A40Z": EOF ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-postgres-566f8bdddc-dzl26.log ==================== ERROR: cannot export logs for pod rhoai-model-registries/model-catalog-postgres-566f8bdddc-dzl26: Error from server: Get "https://10.0.138.210:10250/containerLogs/rhoai-model-registries/model-catalog-postgres-566f8bdddc-dzl26/postgresql?sinceTime=2026-07-20T05%3A45%3A40Z": EOF