==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== redhat-ods-applications/agent-ops-ui: ready=0/1 available=0 redhat-ods-applications/ai-gateway-operator: ready=0/1 available=0 redhat-ods-applications/aihub-controller-manager: ready=0/1 available=0 redhat-ods-applications/automl-ui: ready=0/1 available=0 redhat-ods-applications/autorag-ui: ready=0/1 available=0 redhat-ods-applications/catalog-controller-manager: ready=0/1 available=0 redhat-ods-applications/dashboard-operator: ready=0/1 available=0 redhat-ods-applications/dashboard-redirect: ready=0/2 available=0 redhat-ods-applications/data-science-pipelines-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/eval-hub-ui: ready=0/1 available=0 redhat-ods-applications/feast-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/gen-ai-ui: ready=0/1 available=0 redhat-ods-applications/kserve-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-localmodel-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-trainer-controller-manager: ready=0/1 available=0 redhat-ods-applications/kuberay-operator: ready=0/1 available=0 redhat-ods-applications/llmisvc-controller-manager: ready=0/1 available=0 redhat-ods-applications/maas-controller: ready=0/1 available=0 redhat-ods-applications/maas-ui: ready=0/1 available=0 redhat-ods-applications/mlflow-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/mlflow-ui: ready=0/1 available=0 redhat-ods-applications/model-registry-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-registry-ui: ready=0/1 available=0 redhat-ods-applications/model-serving-api: ready=0/1 available=0 redhat-ods-applications/notebook-controller-deployment: ready=0/1 available=0 redhat-ods-applications/notebooks-ui: ready=0/1 available=0 redhat-ods-applications/odh-model-controller: ready=0/1 available=0 redhat-ods-applications/odh-notebook-controller-manager: ready=0/1 available=0 redhat-ods-applications/odh-observability: ready=0/1 available=0 redhat-ods-applications/ogx-k8s-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/opendatahub-feast-operator: ready=0/1 available=0 redhat-ods-applications/opendatahub-ogx-operator: ready=0/1 available=0 redhat-ods-applications/rhods-dashboard: ready=0/2 available=0 redhat-ods-applications/spark-operator-controller: ready=0/1 available=0 redhat-ods-applications/spark-operator-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook: ready=0/1 available=0 redhat-ods-applications/trainer-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/trustyai-service-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/workbenches-operator: ready=0/1 available=0 redhat-ods-operator/rhods-operator: ready=0/3 available=0 rhoai-model-registries/model-catalog: ready=0/1 available=0 rhoai-model-registries/model-catalog-postgres: ready=0/1 available=0 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/ogx-k8s-operator-controller-manager-metrics-monitor "redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/ogx-k8s-operator-controller-manager-metrics-monitor "redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/ogx-k8s-operator-controller-manager-metrics-monitor "redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/ogx-k8s-operator-controller-manager-metrics-monitor "redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/ogx-k8s-operator-controller-manager-metrics-monitor "redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m53s Warning FailedScheduling pod/agent-ops-ui-69d9689c9-92ftx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/agent-ops-ui-69d9689c9-92ftx no nodes available to schedule pods 33s Warning FailedScheduling pod/agent-ops-ui-69d9689c9-92ftx no nodes available to schedule pods 33s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-58cdd76679-fxqkm no nodes available to schedule pods 37s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-58cdd76679-fxqkm no nodes available to schedule pods 5m53s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-58cdd76679-fxqkm 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/kserve-controller-manager-549db7d5ff-g6kqx no nodes available to schedule pods 5m32s Warning FailedScheduling pod/kserve-controller-manager-549db7d5ff-g6kqx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m54s Warning FailedScheduling pod/kserve-localmodel-controller-manager-5bc445646-vfng7 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/kserve-localmodel-controller-manager-5bc445646-vfng7 no nodes available to schedule pods 5m53s Warning FailedScheduling pod/ai-gateway-operator-74b6dcfcdc-bck56 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/ai-gateway-operator-74b6dcfcdc-bck56 no nodes available to schedule pods 33s Warning FailedScheduling pod/ai-gateway-operator-74b6dcfcdc-bck56 no nodes available to schedule pods 33s Warning FailedScheduling pod/kserve-localmodel-controller-manager-5bc445646-vfng7 no nodes available to schedule pods 81m Warning FailedScheduling pod/kserve-localmodelnode-agent-fr6xv 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 81m Warning FailedScheduling pod/kserve-localmodelnode-agent-vfxwm 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 33s Warning FailedScheduling pod/spark-operator-controller-7558766b8d-tcbfb no nodes available to schedule pods 37s Warning FailedScheduling pod/spark-operator-controller-7558766b8d-tcbfb no nodes available to schedule pods 5m31s Warning FailedScheduling pod/kserve-module-controller-manager-775f5b5bc5-jjls9 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 33s Warning FailedScheduling pod/gen-ai-ui-6dcbf544f6-5l748 no nodes available to schedule pods 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33s Warning FailedScheduling pod/odh-notebook-controller-manager-777b5c4fdc-ptgrq no nodes available to schedule pods 37s Warning FailedScheduling pod/odh-notebook-controller-manager-777b5c4fdc-ptgrq no nodes available to schedule pods 3s Warning FailedScheduling pod/workbenches-operator-5cb6c594f8-kgj4p no nodes available to schedule pods 7s Warning FailedScheduling pod/workbenches-operator-5cb6c594f8-kgj4p no nodes available to schedule pods 5m32s Warning FailedScheduling pod/workbenches-operator-5cb6c594f8-kgj4p 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 14m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m53s Warning FailedScheduling pod/odh-notebook-controller-manager-777b5c4fdc-ptgrq 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 81m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33s Warning FailedScheduling pod/notebooks-ui-5c6747cf85-6s2m9 no nodes available to schedule pods 5m54s Warning FailedScheduling pod/autorag-ui-76b845c4d4-hbds7 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/autorag-ui-76b845c4d4-hbds7 no nodes available to schedule pods 33s Warning FailedScheduling pod/autorag-ui-76b845c4d4-hbds7 no nodes available to schedule pods 37s Warning FailedScheduling pod/notebooks-ui-5c6747cf85-6s2m9 no nodes available to schedule pods 5m53s Warning FailedScheduling pod/notebooks-ui-5c6747cf85-6s2m9 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 14m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m32s Warning FailedScheduling pod/catalog-controller-manager-5dcddfb9cc-stxb6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/catalog-controller-manager-5dcddfb9cc-stxb6 no nodes available to schedule pods 3s Warning FailedScheduling pod/catalog-controller-manager-5dcddfb9cc-stxb6 no nodes available to schedule pods 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3s Warning FailedScheduling pod/model-registry-operator-controller-manager-68589b648b-ffdht no nodes available to schedule pods 7s Warning FailedScheduling pod/model-registry-operator-controller-manager-68589b648b-ffdht no nodes available to schedule pods 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3s Warning FailedScheduling pod/kserve-controller-manager-549db7d5ff-g6kqx no nodes available to schedule pods 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m31s Warning FailedScheduling pod/dashboard-operator-64d7c8d8db-49k5n 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/dashboard-operator-64d7c8d8db-49k5n no nodes available to schedule pods 3s Warning FailedScheduling pod/dashboard-operator-64d7c8d8db-49k5n no nodes available to schedule pods 5m31s Warning FailedScheduling pod/model-registry-operator-controller-manager-68589b648b-ffdht 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 33s Warning FailedScheduling pod/mlflow-ui-9758df657-l8vmz no nodes available to schedule pods 37s Warning FailedScheduling pod/mlflow-ui-9758df657-l8vmz no nodes available to schedule pods 5m55s Warning FailedScheduling pod/mlflow-ui-9758df657-l8vmz 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 3s Warning FailedScheduling pod/mlflow-operator-controller-manager-fbd674d66-m88rh no nodes available to schedule pods 7s Warning FailedScheduling pod/mlflow-operator-controller-manager-fbd674d66-m88rh no nodes available to schedule pods 5m32s Warning FailedScheduling pod/mlflow-operator-controller-manager-fbd674d66-m88rh 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m31s Warning FailedScheduling pod/rhods-dashboard-86644c7f4d-hg8q7 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/rhods-dashboard-86644c7f4d-hg8q7 no nodes available to schedule pods 33s Warning FailedScheduling pod/trustyai-service-operator-controller-manager-8dddd65c6-4zpg4 no nodes available to schedule pods 3s Warning FailedScheduling pod/rhods-dashboard-86644c7f4d-hg8q7 no nodes available to schedule pods 5m33s Warning FailedScheduling pod/llmisvc-controller-manager-6cf9b4f89c-lfcsl 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m54s Warning FailedScheduling pod/rhods-dashboard-86644c7f4d-rf6gt 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/trustyai-service-operator-controller-manager-8dddd65c6-4zpg4 no nodes available to schedule pods 5m54s Warning FailedScheduling pod/trustyai-service-operator-controller-manager-8dddd65c6-4zpg4 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m32s Warning FailedScheduling pod/dashboard-redirect-576b457675-sx4mh 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/dashboard-redirect-576b457675-sx4mh no nodes available to schedule pods 3s Warning FailedScheduling pod/dashboard-redirect-576b457675-sx4mh no nodes available to schedule pods 37s Warning FailedScheduling pod/rhods-dashboard-86644c7f4d-rf6gt no nodes available to schedule pods 33s Warning FailedScheduling pod/rhods-dashboard-86644c7f4d-rf6gt no nodes available to schedule pods 3s Warning FailedScheduling pod/llmisvc-controller-manager-6cf9b4f89c-lfcsl no nodes available to schedule pods 7s Warning FailedScheduling pod/llmisvc-controller-manager-6cf9b4f89c-lfcsl no nodes available to schedule pods 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3s Warning FailedScheduling pod/trainer-operator-controller-manager-5dd7dfcb56-wbnx2 no nodes available to schedule pods 5m31s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-7847455mw242 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-7847455mw242 no nodes available to schedule pods 3s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-7847455mw242 no nodes available to schedule pods 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33s Warning FailedScheduling pod/maas-controller-7ddc487d54-pjc6b no nodes available to schedule pods 37s Warning FailedScheduling pod/maas-controller-7ddc487d54-pjc6b no nodes available to schedule pods 5m54s Warning FailedScheduling pod/maas-controller-7ddc487d54-pjc6b 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/trainer-operator-controller-manager-5dd7dfcb56-wbnx2 no nodes available to schedule pods 5m54s Warning FailedScheduling pod/eval-hub-ui-55bcbd4779-tzkrh 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/eval-hub-ui-55bcbd4779-tzkrh no nodes available to schedule pods 33s Warning FailedScheduling pod/eval-hub-ui-55bcbd4779-tzkrh no nodes available to schedule pods 5m32s Warning FailedScheduling pod/trainer-operator-controller-manager-5dd7dfcb56-wbnx2 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m30s Warning FailedScheduling pod/feast-operator-controller-manager-fcff4ddc6-gc74x 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/feast-operator-controller-manager-fcff4ddc6-gc74x no nodes available to schedule pods 2s Warning FailedScheduling pod/feast-operator-controller-manager-fcff4ddc6-gc74x no nodes available to schedule pods 33s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-696cfc9d78-mc2bv no nodes available to schedule pods 37s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-696cfc9d78-mc2bv no nodes available to schedule pods 5m54s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-696cfc9d78-mc2bv 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m54s Warning FailedScheduling pod/spark-operator-controller-7558766b8d-tcbfb 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 3s Warning FailedScheduling pod/kserve-module-controller-manager-775f5b5bc5-jjls9 no nodes available to schedule pods 7s Warning FailedScheduling pod/kserve-module-controller-manager-775f5b5bc5-jjls9 no nodes available to schedule pods 5m53s Warning FailedScheduling pod/gen-ai-ui-6dcbf544f6-5l748 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/gen-ai-ui-6dcbf544f6-5l748 no nodes available to schedule pods 117m Warning FailedMount pod/odh-observability-dbfff9446-t6d6x MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 117m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 116m Warning FailedMount pod/kuberay-operator-668c5f59bd-nbscf MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 116m Warning BadConfig certificaterequest/serving-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 116m Warning FailedMount pod/model-registry-ui-85798dff55-fxb57 MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 116m Warning FailedMount pod/notebooks-ui-5c6747cf85-gjzm7 MountVolume.SetUp failed for volume "proxy-tls" : secret "notebooks-proxy-tls" not found 115m Warning Unhealthy pod/rhods-dashboard-6fcf89d9cd-bpq8m Readiness probe errored and resulted in unknown state: rpc error: code = Canceled desc = context canceled 115m Warning Unhealthy pod/rhods-dashboard-6fcf89d9cd-bpq8m Readiness probe failed: Get "https://10.133.0.48:8943/healthcheck": dial tcp 10.133.0.48:8943: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-6fcf89d9cd-bpq8m Readiness probe failed: Get "https://10.133.0.48:8444/healthz": dial tcp 10.133.0.48:8444: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-6fcf89d9cd-rbcnb Readiness probe errored and resulted in unknown state: rpc error: code = Canceled desc = context canceled 115m Warning Unhealthy pod/rhods-dashboard-6fcf89d9cd-rbcnb Readiness probe failed: Get "https://10.134.0.47:8943/healthcheck": dial tcp 10.134.0.47:8943: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-6fcf89d9cd-rbcnb Readiness probe failed: Get "https://10.134.0.47:8444/healthz": dial tcp 10.134.0.47:8444: connect: connection refused 81m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 81m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 22m Warning Unhealthy pod/mlflow-77b77b7789-qrvsw Readiness probe failed: Get "https://10.133.0.116:8443/mlflow/api/3.0/mlflow/server-info": dial tcp 10.133.0.116:8443: connect: connection refused 16m Warning Unhealthy pod/mlflow-8995d7b65-vtgb4 Readiness probe failed: Get "https://10.133.0.119:8443/mlflow/api/3.0/mlflow/server-info": dial tcp 10.133.0.119:8443: connect: connection refused 7m50s Warning Unhealthy pod/dashboard-operator-64d7c8d8db-qm6w6 Readiness probe failed: Get "http://10.132.0.51:8081/readyz": dial tcp 10.132.0.51:8081: connect: connection refused 7m4s Warning NodeNotReady pod/odh-notebook-controller-manager-777b5c4fdc-gcf4g Node is not ready 7m4s Warning NodeNotReady pod/kserve-localmodelnode-agent-fr6xv Node is not ready 7m4s Warning NodeNotReady pod/gen-ai-ui-6dcbf544f6-gtljh Node is not ready 7m4s Warning NodeNotReady pod/kserve-localmodel-controller-manager-5bc445646-fszr7 Node is not ready 7m4s Warning NodeNotReady pod/maas-ui-d874967cc-k49d8 Node is not ready 7m4s Warning NodeNotReady pod/kserve-module-controller-manager-775f5b5bc5-kdgd8 Node is not ready 7m4s Warning NodeNotReady pod/mlflow-operator-controller-manager-fbd674d66-f8fxp Node is not ready 7m4s Warning NodeNotReady pod/autorag-ui-76b845c4d4-7rqdl Node is not ready 7m4s Warning NodeNotReady pod/kserve-localmodelnode-agent-vfxwm Node is not ready 7m4s Warning NodeNotReady pod/ogx-k8s-operator-controller-manager-58cdd76679-n58wz Node is not ready 7m4s Warning NodeNotReady pod/spark-operator-webhook-7f794cfbb7-sk7fg Node is not ready 7m4s Warning NodeNotReady pod/notebooks-ui-5c6747cf85-gjzm7 Node is not ready 7m4s Warning NodeNotReady pod/catalog-controller-manager-5dcddfb9cc-hb8wk Node is not ready 7m4s Warning NodeNotReady pod/mlflow-ui-9758df657-4sjln Node is not ready 7m3s Warning NodeNotReady pod/trainer-operator-controller-manager-5dd7dfcb56-s569b Node is not ready 7m3s Warning NodeNotReady pod/opendatahub-feast-operator-6d65f59c96-tpdhf Node is not ready 7m3s Warning NodeNotReady pod/kuberay-operator-668c5f59bd-nbscf Node is not ready 7m3s Warning NodeNotReady pod/opendatahub-ogx-operator-86649f78d7-ltpbg Node is not ready 7m3s Warning NodeNotReady pod/kubeflow-trainer-controller-manager-696cfc9d78-cqjdc Node is not ready 7m3s Warning NodeNotReady pod/odh-model-controller-849d64c4b4-t2bsl Node is not ready 7m3s Warning NodeNotReady pod/kserve-controller-manager-549db7d5ff-df684 Node is not ready 7m3s Warning NodeNotReady pod/agent-ops-ui-69d9689c9-48qms Node is not ready 7m3s Warning NodeNotReady pod/llmisvc-controller-manager-6cf9b4f89c-s8j76 Node is not ready 7m3s Warning NodeNotReady pod/dashboard-operator-64d7c8d8db-qm6w6 Node is not ready 7m3s Warning NodeNotReady pod/notebook-controller-deployment-db7d4bdbf-545g7 Node is not ready 7m3s Warning NodeNotReady pod/rhods-dashboard-86644c7f4d-qjlpz Node is not ready 7m3s Warning NodeNotReady pod/feast-operator-controller-manager-fcff4ddc6-nb2mr Node is not ready 7m3s Warning NodeNotReady pod/model-registry-ui-85798dff55-fxb57 Node is not ready 7m3s Warning NodeNotReady pod/dashboard-redirect-576b457675-hfd6j Node is not ready 7m3s Warning NodeNotReady pod/eval-hub-ui-55bcbd4779-qdtf5 Node is not ready 7m3s Warning NodeNotReady pod/data-science-pipelines-operator-controller-manager-7847455bnmgk Node is not ready 7m3s Warning NodeNotReady pod/model-registry-operator-controller-manager-68589b648b-5bh5j Node is not ready 7m3s Warning NodeNotReady pod/trustyai-service-operator-controller-manager-8dddd65c6-rz5zx Node is not ready 7m3s Warning NodeNotReady pod/aihub-controller-manager-85bff87dd6-znlnr Node is not ready 7m3s Warning NodeNotReady pod/rhods-dashboard-86644c7f4d-t9w6n Node is not ready 7m3s Warning NodeNotReady pod/workbenches-operator-5cb6c594f8-t7dz4 Node is not ready 7m3s Warning NodeNotReady pod/ai-gateway-operator-74b6dcfcdc-7pzlr Node is not ready 7m2s Warning NodeNotReady pod/dashboard-redirect-576b457675-5b4z2 Node is not ready 7m2s Warning NodeNotReady pod/automl-ui-78945f54f8-vmbs5 Node is not ready 7m2s Warning NodeNotReady pod/spark-operator-module-controller-manager-565f8d557c-l67wm Node is not ready 7m2s Warning NodeNotReady pod/spark-operator-controller-7558766b8d-rl2hr Node is not ready 7m2s Warning NodeNotReady pod/maas-controller-7ddc487d54-n6kd7 Node is not ready 7m2s Warning NodeNotReady pod/odh-observability-dbfff9446-t6d6x Node is not ready 7m2s Warning NodeNotReady pod/model-serving-api-8445b5bcb7-d548m Node is not ready 5m46s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: skipping Pod rhods-dashboard-86644c7f4d-t9w6n for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-136-8.ec2.internal Not Found 61s Warning FailedCreate replicaset/aihub-controller-manager-85bff87dd6 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 61s Warning FailedCreate replicaset/spark-operator-webhook-7f794cfbb7 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 60s Warning FailedCreate replicaset/opendatahub-ogx-operator-86649f78d7 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 59s Warning FailedCreate replicaset/notebook-controller-deployment-db7d4bdbf Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 59s Warning FailedCreate replicaset/odh-model-controller-849d64c4b4 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 56s Warning FailedCreate replicaset/automl-ui-78945f54f8 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/maas-ui-d874967cc Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 45s Warning FailedCreate replicaset/kuberay-operator-668c5f59bd Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 44s Warning FailedCreate replicaset/model-serving-api-8445b5bcb7 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 44s Warning FailedCreate replicaset/spark-operator-module-controller-manager-565f8d557c Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 43s Warning FailedCreate replicaset/odh-observability-dbfff9446 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 43s Warning FailedCreate replicaset/opendatahub-feast-operator-6d65f59c96 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 43s Warning FailedCreate replicaset/model-registry-ui-85798dff55 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 8s Warning FailedCreate replicaset/dashboard-redirect-576b457675 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 4s Warning FailedScheduling pod/rhods-operator-6d7db7fb99-qhrvv no nodes available to schedule pods 5m53s Warning FailedScheduling pod/rhods-operator-6d7db7fb99-bl9kl 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 38s Warning FailedScheduling pod/rhods-operator-6d7db7fb99-bl9kl no nodes available to schedule pods 34s Warning FailedScheduling pod/rhods-operator-6d7db7fb99-bl9kl no nodes available to schedule pods 5m31s Warning FailedScheduling pod/rhods-operator-6d7db7fb99-qhrvv 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 8s Warning FailedScheduling pod/rhods-operator-6d7db7fb99-qhrvv no nodes available to schedule pods 117m Warning AppliedWithWarnings installplan/install-xn5bt 1 warning(s) generated during installation of operator "rhods-operator.3.6.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 7m5s Warning NodeNotReady pod/rhods-operator-6d7db7fb99-fsxnj Node is not ready 7m4s Warning NodeNotReady pod/rhods-operator-6d7db7fb99-k9njp Node is not ready 7m3s Warning NodeNotReady pod/rhods-operator-6d7db7fb99-qbxmv Node is not ready 7m3s Warning ComponentUnhealthy clusterserviceversion/rhods-operator.3.6.0-ea.2 installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. 5m45s Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: skipping Pod rhods-operator-6d7db7fb99-fsxnj for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-136-8.ec2.internal Not Found 5m45s Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: skipping Pod rhods-operator-6d7db7fb99-fsxnj for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-136-8.ec2.internal Not Found 119s Warning InstallCheckFailed clusterserviceversion/rhods-operator.3.6.0-ea.2 install timeout 8s Warning FailedCreate replicaset/rhods-operator-6d7db7fb99 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 8s Warning FailedScheduling pod/model-catalog-postgres-69664bc65d-gwp7p no nodes available to schedule pods 83m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 83m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 85m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 5m32s Warning FailedScheduling pod/model-catalog-postgres-69664bc65d-gwp7p 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 4s Warning FailedScheduling pod/model-catalog-postgres-69664bc65d-gwp7p no nodes available to schedule pods 4s Warning FailedScheduling pod/model-catalog-6c54df8856-j9ffq no nodes available to schedule pods 8s Warning FailedScheduling pod/model-catalog-6c54df8856-j9ffq no nodes available to schedule pods 5m33s Warning FailedScheduling pod/model-catalog-6c54df8856-j9ffq 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 116m Warning FailedMount pod/model-catalog-6c54df8856-m4dlg MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 116m Warning Unhealthy pod/model-catalog-6c54df8856-m4dlg Readiness probe failed: Get "http://10.132.0.61:8080/readyz": dial tcp 10.132.0.61:8080: connect: connection refused 85m Warning FailedMount pod/model-registry0-797fd97cf6-hwg7w MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-registry0-kube-rbac-proxy-config" not found 7m4s Warning NodeNotReady pod/model-catalog-postgres-69664bc65d-fv7x8 Node is not ready 7m4s Warning NodeNotReady pod/model-catalog-6c54df8856-m4dlg Node is not ready (no matching error/webhook lines in operator logs for this time window) --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-09-15T11:25:42Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.22.12 Kubernetes Version: v1.35.6 --- Namespaces --- namespace/opendatahub namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION RELEASE REPLACES PHASE authorino-operator.v1.4.3 Authorino Operator 1.4.3 authorino-operator.v1.4.2 Installing cert-manager-operator.v1.20.0 cert-manager Operator for Red Hat OpenShift 1.20.0 cert-manager-operator.v1.19.0 Installing custom-metrics-autoscaler.v2.19.0-3 Custom Metrics Autoscaler 2.19.0-3 custom-metrics-autoscaler.v2.19.0-2 Installing dns-operator.v1.4.2 DNS Operator 1.4.2 dns-operator.v1.4.1 Installing kueue-operator.v1.4.2 Red Hat build of Kueue 1.4.2 Installing limitador-operator.v1.4.2 Limitador Operator 1.4.2 limitador-operator.v1.4.1 Installing rhcl-operator.v1.4.3 Red Hat Connectivity Link 1.4.3 rhcl-operator.v1.4.2 Installing rhods-operator.3.6.0-ea.2 Red Hat OpenShift AI 3.6.0-ea.2 Installing --- CSVs not in Succeeded phase --- authorino-operator.v1.4.3: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace cert-manager-operator.v1.20.0: phase=Installing reason=Copied message=The operator is running in cert-manager-operator but is managing this namespace custom-metrics-autoscaler.v2.19.0-3: phase=Installing reason=Copied message=The operator is running in openshift-keda but is managing this namespace dns-operator.v1.4.2: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace kueue-operator.v1.4.2: phase=Installing reason=Copied message=The operator is running in openshift-kueue-operator but is managing this namespace limitador-operator.v1.4.2: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace rhcl-operator.v1.4.3: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace rhods-operator.3.6.0-ea.2: phase=Installing reason=InstallWaiting message=installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aigateway: managementState=Managed aipipelines: managementState=Managed batchGateway: managementState=Removed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Removed llamastackoperator: managementState=n/a mcplifecycleoperator: managementState=Removed mlflowoperator: managementState=Managed modelregistry: managementState=Managed modelsAsAService: managementState=Managed ogx: managementState=Managed ray: managementState=Managed sparkoperator: managementState=Managed trainer: managementState=Managed trainingoperator: managementState=Removed trustyai: managementState=Managed workbenches: managementState=Managed workbenchesV2: managementState=Removed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= ModulesReady: status=True reason= message= AIGatewayReady: status=True reason= message= AIPipelinesReady: status=True reason= message= BatchGatewayReady: status=False reason=Removed message=Submodule ManagementState is set to Removed DashboardReady: status=True reason=AllDependentsHealthy message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=False reason=Removed message=Component ManagementState is set to Removed MCPLifecycleOperatorReady: status=False reason=Removed message=Module ManagementState is set to Removed MLflowOperatorReady: status=True reason=Ready message=MLflowOperator is ready to manage MLflow custom resources ModelRegistryReady: status=True reason=AllDependentsHealthy message= ModelsAsAServiceReady: status=True reason=Ready message=modelsAsAService is Managed and deployments are available OGXReady: status=True reason=AllDependentsHealthy message= RayReady: status=True reason= message= SparkOperatorReady: status=True reason=AllDependentsHealthy message= TrainerReady: status=True reason= message= TrustyAIReady: status=True reason= message= WorkbenchesReady: status=True reason=ReconcileSuccess message=Workbenches component is ready WorkbenchesV2Ready: status=False reason=Removed message=Submodule ManagementState is set to Removed --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 117m Ready 2026-09-15T09:28:27Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=True reason=Ready message=Monitoring stack is initialized MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in Monitoring CR OpenTelemetryCollectorAvailable: status=False reason=MetricsAndTracesNotConfigured message=Metrics and traces are not configured in Monitoring CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in Monitoring CR NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR PersesAvailable: status=False reason=MetricsAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured ProvisioningSucceeded: status=True reason=Available message= Ready: status=True reason=Available message= --- Component CRs (Ready) --- NAME READY REASON VERSION aigateway.components.platform.opendatahub.io/default-aigateway True 1.26.2 NAME APPNAMESPACE AGE aihub.components.platform.opendatahub.io/default-aihub redhat-ods-applications 116m NAME PHASE READY REASON URL AGE dashboard.components.platform.opendatahub.io/default-dashboard Ready True AllDependentsHealthy https://rh-ai.apps.4ca33028e2d172965d05.prod.konflux-ocp-ci.dev/ 116m NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON mlflowoperator.components.platform.opendatahub.io/default-mlflowoperator True Ready NAME READY REASON DISTRIBUTION ogx.components.platform.opendatahub.io/default-ogx True AllDependentsHealthy Red Hat OpenShift AI NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON sparkoperator.components.platform.opendatahub.io/default-sparkoperator True AllDependentsHealthy NAME READY REASON AGE trainer.components.platform.opendatahub.io/default-trainer True 116m NAME READY REASON trustyai.components.platform.opendatahub.io/default-trustyai True NAME READY REASON PHASE workbenches.components.platform.opendatahub.io/default-workbenches True ReconcileSuccess Ready --- Deployments not fully ready (ODH/RHOAI namespaces) --- redhat-ods-applications/agent-ops-ui: ready=0/1 available=0 redhat-ods-applications/ai-gateway-operator: ready=0/1 available=0 redhat-ods-applications/aihub-controller-manager: ready=0/1 available=0 redhat-ods-applications/automl-ui: ready=0/1 available=0 redhat-ods-applications/autorag-ui: ready=0/1 available=0 redhat-ods-applications/catalog-controller-manager: ready=0/1 available=0 redhat-ods-applications/dashboard-operator: ready=0/1 available=0 redhat-ods-applications/dashboard-redirect: ready=0/2 available=0 redhat-ods-applications/data-science-pipelines-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/eval-hub-ui: ready=0/1 available=0 redhat-ods-applications/feast-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/gen-ai-ui: ready=0/1 available=0 redhat-ods-applications/kserve-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-localmodel-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-trainer-controller-manager: ready=0/1 available=0 redhat-ods-applications/kuberay-operator: ready=0/1 available=0 redhat-ods-applications/llmisvc-controller-manager: ready=0/1 available=0 redhat-ods-applications/maas-controller: ready=0/1 available=0 redhat-ods-applications/maas-ui: ready=0/1 available=0 redhat-ods-applications/mlflow-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/mlflow-ui: ready=0/1 available=0 redhat-ods-applications/model-registry-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-registry-ui: ready=0/1 available=0 redhat-ods-applications/model-serving-api: ready=0/1 available=0 redhat-ods-applications/notebook-controller-deployment: ready=0/1 available=0 redhat-ods-applications/notebooks-ui: ready=0/1 available=0 redhat-ods-applications/odh-model-controller: ready=0/1 available=0 redhat-ods-applications/odh-notebook-controller-manager: ready=0/1 available=0 redhat-ods-applications/odh-observability: ready=0/1 available=0 redhat-ods-applications/ogx-k8s-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/opendatahub-feast-operator: ready=0/1 available=0 redhat-ods-applications/opendatahub-ogx-operator: ready=0/1 available=0 redhat-ods-applications/rhods-dashboard: ready=0/2 available=0 redhat-ods-applications/spark-operator-controller: ready=0/1 available=0 redhat-ods-applications/spark-operator-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook: ready=0/1 available=0 redhat-ods-applications/trainer-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/trustyai-service-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/workbenches-operator: ready=0/1 available=0 redhat-ods-operator/rhods-operator: ready=0/3 available=0 rhoai-model-registries/model-catalog: ready=0/1 available=0 rhoai-model-registries/model-catalog-postgres: ready=0/1 available=0 --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- redhat-ods-applications/agent-ops-ui-69d9689c9-92ftx: phase=Pending reason= redhat-ods-applications/ai-gateway-operator-74b6dcfcdc-bck56: phase=Pending reason= redhat-ods-applications/autorag-ui-76b845c4d4-hbds7: phase=Pending reason= redhat-ods-applications/catalog-controller-manager-5dcddfb9cc-stxb6: phase=Pending reason= redhat-ods-applications/dashboard-operator-64d7c8d8db-49k5n: phase=Pending reason= redhat-ods-applications/dashboard-redirect-576b457675-sx4mh: phase=Pending reason= redhat-ods-applications/data-science-pipelines-operator-controller-manager-7847455mw242: phase=Pending reason= redhat-ods-applications/eval-hub-ui-55bcbd4779-tzkrh: phase=Pending reason= redhat-ods-applications/feast-operator-controller-manager-fcff4ddc6-gc74x: phase=Pending reason= redhat-ods-applications/gen-ai-ui-6dcbf544f6-5l748: phase=Pending reason= redhat-ods-applications/kserve-controller-manager-549db7d5ff-g6kqx: phase=Pending reason= redhat-ods-applications/kserve-localmodel-controller-manager-5bc445646-vfng7: phase=Pending reason= redhat-ods-applications/kserve-module-controller-manager-775f5b5bc5-jjls9: phase=Pending reason= redhat-ods-applications/kubeflow-trainer-controller-manager-696cfc9d78-mc2bv: phase=Pending reason= redhat-ods-applications/llmisvc-controller-manager-6cf9b4f89c-lfcsl: phase=Pending reason= redhat-ods-applications/maas-controller-7ddc487d54-pjc6b: phase=Pending reason= redhat-ods-applications/mlflow-operator-controller-manager-fbd674d66-m88rh: phase=Pending reason= redhat-ods-applications/mlflow-ui-9758df657-l8vmz: phase=Pending reason= redhat-ods-applications/model-registry-operator-controller-manager-68589b648b-ffdht: phase=Pending reason= redhat-ods-applications/notebooks-ui-5c6747cf85-6s2m9: phase=Pending reason= redhat-ods-applications/odh-notebook-controller-manager-777b5c4fdc-ptgrq: phase=Pending reason= redhat-ods-applications/ogx-k8s-operator-controller-manager-58cdd76679-fxqkm: phase=Pending reason= redhat-ods-applications/rhods-dashboard-86644c7f4d-hg8q7: phase=Pending reason= redhat-ods-applications/rhods-dashboard-86644c7f4d-rf6gt: phase=Pending reason= redhat-ods-applications/spark-operator-controller-7558766b8d-tcbfb: phase=Pending reason= redhat-ods-applications/trainer-operator-controller-manager-5dd7dfcb56-wbnx2: phase=Pending reason= redhat-ods-applications/trustyai-service-operator-controller-manager-8dddd65c6-4zpg4: phase=Pending reason= redhat-ods-applications/workbenches-operator-5cb6c594f8-kgj4p: phase=Pending reason= redhat-ods-operator/rhods-operator-6d7db7fb99-bl9kl: phase=Pending reason= redhat-ods-operator/rhods-operator-6d7db7fb99-qhrvv: phase=Pending reason= rhoai-model-registries/model-catalog-6c54df8856-j9ffq: phase=Pending reason= rhoai-model-registries/model-catalog-postgres-69664bc65d-gwp7p: phase=Pending reason= --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/opendatahub --- (no events) --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/ogx-k8s-operator-controller-manager-metrics-monitor "redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/ogx-k8s-operator-controller-manager-metrics-monitor "redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/ogx-k8s-operator-controller-manager-metrics-monitor "redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/ogx-k8s-operator-controller-manager-metrics-monitor "redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/ogx-k8s-operator-controller-manager-metrics-monitor "redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m53s Warning FailedScheduling pod/agent-ops-ui-69d9689c9-92ftx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/agent-ops-ui-69d9689c9-92ftx no nodes available to schedule pods 33s Warning FailedScheduling pod/agent-ops-ui-69d9689c9-92ftx no nodes available to schedule pods 33s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-58cdd76679-fxqkm no nodes available to schedule pods 37s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-58cdd76679-fxqkm no nodes available to schedule pods 5m53s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-58cdd76679-fxqkm 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/kserve-controller-manager-549db7d5ff-g6kqx no nodes available to schedule pods 5m32s Warning FailedScheduling pod/kserve-controller-manager-549db7d5ff-g6kqx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m54s Warning FailedScheduling pod/kserve-localmodel-controller-manager-5bc445646-vfng7 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/kserve-localmodel-controller-manager-5bc445646-vfng7 no nodes available to schedule pods 5m53s Warning FailedScheduling pod/ai-gateway-operator-74b6dcfcdc-bck56 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/ai-gateway-operator-74b6dcfcdc-bck56 no nodes available to schedule pods 33s Warning FailedScheduling pod/ai-gateway-operator-74b6dcfcdc-bck56 no nodes available to schedule pods 33s Warning FailedScheduling pod/kserve-localmodel-controller-manager-5bc445646-vfng7 no nodes available to schedule pods 81m Warning FailedScheduling pod/kserve-localmodelnode-agent-fr6xv 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 81m Warning FailedScheduling pod/kserve-localmodelnode-agent-vfxwm 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 33s Warning FailedScheduling pod/spark-operator-controller-7558766b8d-tcbfb no nodes available to schedule pods 37s Warning FailedScheduling pod/spark-operator-controller-7558766b8d-tcbfb no nodes available to schedule pods 5m31s Warning FailedScheduling pod/kserve-module-controller-manager-775f5b5bc5-jjls9 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 33s Warning FailedScheduling pod/gen-ai-ui-6dcbf544f6-5l748 no nodes available to schedule pods 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33s Warning FailedScheduling pod/odh-notebook-controller-manager-777b5c4fdc-ptgrq no nodes available to schedule pods 37s Warning FailedScheduling pod/odh-notebook-controller-manager-777b5c4fdc-ptgrq no nodes available to schedule pods 3s Warning FailedScheduling pod/workbenches-operator-5cb6c594f8-kgj4p no nodes available to schedule pods 7s Warning FailedScheduling pod/workbenches-operator-5cb6c594f8-kgj4p no nodes available to schedule pods 5m32s Warning FailedScheduling pod/workbenches-operator-5cb6c594f8-kgj4p 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 14m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m53s Warning FailedScheduling pod/odh-notebook-controller-manager-777b5c4fdc-ptgrq 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/trustyai-service-operator-service-monitor "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 81m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33s Warning FailedScheduling pod/notebooks-ui-5c6747cf85-6s2m9 no nodes available to schedule pods 5m54s Warning FailedScheduling pod/autorag-ui-76b845c4d4-hbds7 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/autorag-ui-76b845c4d4-hbds7 no nodes available to schedule pods 33s Warning FailedScheduling pod/autorag-ui-76b845c4d4-hbds7 no nodes available to schedule pods 37s Warning FailedScheduling pod/notebooks-ui-5c6747cf85-6s2m9 no nodes available to schedule pods 5m53s Warning FailedScheduling pod/notebooks-ui-5c6747cf85-6s2m9 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 14m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m32s Warning FailedScheduling pod/catalog-controller-manager-5dcddfb9cc-stxb6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/catalog-controller-manager-5dcddfb9cc-stxb6 no nodes available to schedule pods 3s Warning FailedScheduling pod/catalog-controller-manager-5dcddfb9cc-stxb6 no nodes available to schedule pods 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3s Warning FailedScheduling pod/model-registry-operator-controller-manager-68589b648b-ffdht no nodes available to schedule pods 7s Warning FailedScheduling pod/model-registry-operator-controller-manager-68589b648b-ffdht no nodes available to schedule pods 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3s Warning FailedScheduling pod/kserve-controller-manager-549db7d5ff-g6kqx no nodes available to schedule pods 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m31s Warning FailedScheduling pod/dashboard-operator-64d7c8d8db-49k5n 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/dashboard-operator-64d7c8d8db-49k5n no nodes available to schedule pods 3s Warning FailedScheduling pod/dashboard-operator-64d7c8d8db-49k5n no nodes available to schedule pods 5m31s Warning FailedScheduling pod/model-registry-operator-controller-manager-68589b648b-ffdht 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 33s Warning FailedScheduling pod/mlflow-ui-9758df657-l8vmz no nodes available to schedule pods 37s Warning FailedScheduling pod/mlflow-ui-9758df657-l8vmz no nodes available to schedule pods 5m55s Warning FailedScheduling pod/mlflow-ui-9758df657-l8vmz 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 3s Warning FailedScheduling pod/mlflow-operator-controller-manager-fbd674d66-m88rh no nodes available to schedule pods 7s Warning FailedScheduling pod/mlflow-operator-controller-manager-fbd674d66-m88rh no nodes available to schedule pods 5m32s Warning FailedScheduling pod/mlflow-operator-controller-manager-fbd674d66-m88rh 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m31s Warning FailedScheduling pod/rhods-dashboard-86644c7f4d-hg8q7 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/rhods-dashboard-86644c7f4d-hg8q7 no nodes available to schedule pods 33s Warning FailedScheduling pod/trustyai-service-operator-controller-manager-8dddd65c6-4zpg4 no nodes available to schedule pods 3s Warning FailedScheduling pod/rhods-dashboard-86644c7f4d-hg8q7 no nodes available to schedule pods 5m33s Warning FailedScheduling pod/llmisvc-controller-manager-6cf9b4f89c-lfcsl 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m54s Warning FailedScheduling pod/rhods-dashboard-86644c7f4d-rf6gt 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/trustyai-service-operator-controller-manager-8dddd65c6-4zpg4 no nodes available to schedule pods 5m54s Warning FailedScheduling pod/trustyai-service-operator-controller-manager-8dddd65c6-4zpg4 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m32s Warning FailedScheduling pod/dashboard-redirect-576b457675-sx4mh 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/dashboard-redirect-576b457675-sx4mh no nodes available to schedule pods 3s Warning FailedScheduling pod/dashboard-redirect-576b457675-sx4mh no nodes available to schedule pods 37s Warning FailedScheduling pod/rhods-dashboard-86644c7f4d-rf6gt no nodes available to schedule pods 33s Warning FailedScheduling pod/rhods-dashboard-86644c7f4d-rf6gt no nodes available to schedule pods 3s Warning FailedScheduling pod/llmisvc-controller-manager-6cf9b4f89c-lfcsl no nodes available to schedule pods 7s Warning FailedScheduling pod/llmisvc-controller-manager-6cf9b4f89c-lfcsl no nodes available to schedule pods 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3s Warning FailedScheduling pod/trainer-operator-controller-manager-5dd7dfcb56-wbnx2 no nodes available to schedule pods 5m31s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-7847455mw242 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-7847455mw242 no nodes available to schedule pods 3s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-7847455mw242 no nodes available to schedule pods 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33s Warning FailedScheduling pod/maas-controller-7ddc487d54-pjc6b no nodes available to schedule pods 37s Warning FailedScheduling pod/maas-controller-7ddc487d54-pjc6b no nodes available to schedule pods 5m54s Warning FailedScheduling pod/maas-controller-7ddc487d54-pjc6b 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/trainer-operator-controller-manager-5dd7dfcb56-wbnx2 no nodes available to schedule pods 5m54s Warning FailedScheduling pod/eval-hub-ui-55bcbd4779-tzkrh 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/eval-hub-ui-55bcbd4779-tzkrh no nodes available to schedule pods 33s Warning FailedScheduling pod/eval-hub-ui-55bcbd4779-tzkrh no nodes available to schedule pods 5m32s Warning FailedScheduling pod/trainer-operator-controller-manager-5dd7dfcb56-wbnx2 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m30s Warning FailedScheduling pod/feast-operator-controller-manager-fcff4ddc6-gc74x 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7s Warning FailedScheduling pod/feast-operator-controller-manager-fcff4ddc6-gc74x no nodes available to schedule pods 2s Warning FailedScheduling pod/feast-operator-controller-manager-fcff4ddc6-gc74x no nodes available to schedule pods 33s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-696cfc9d78-mc2bv no nodes available to schedule pods 37s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-696cfc9d78-mc2bv no nodes available to schedule pods 5m54s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-696cfc9d78-mc2bv 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m54s Warning FailedScheduling pod/spark-operator-controller-7558766b8d-tcbfb 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 3s Warning FailedScheduling pod/kserve-module-controller-manager-775f5b5bc5-jjls9 no nodes available to schedule pods 7s Warning FailedScheduling pod/kserve-module-controller-manager-775f5b5bc5-jjls9 no nodes available to schedule pods 5m53s Warning FailedScheduling pod/gen-ai-ui-6dcbf544f6-5l748 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 37s Warning FailedScheduling pod/gen-ai-ui-6dcbf544f6-5l748 no nodes available to schedule pods 117m Warning FailedMount pod/odh-observability-dbfff9446-t6d6x MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 117m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 116m Warning FailedMount pod/kuberay-operator-668c5f59bd-nbscf MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 116m Warning BadConfig certificaterequest/serving-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 116m Warning FailedMount pod/model-registry-ui-85798dff55-fxb57 MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 116m Warning FailedMount pod/notebooks-ui-5c6747cf85-gjzm7 MountVolume.SetUp failed for volume "proxy-tls" : secret "notebooks-proxy-tls" not found 115m Warning Unhealthy pod/rhods-dashboard-6fcf89d9cd-bpq8m Readiness probe errored and resulted in unknown state: rpc error: code = Canceled desc = context canceled 115m Warning Unhealthy pod/rhods-dashboard-6fcf89d9cd-bpq8m Readiness probe failed: Get "https://10.133.0.48:8943/healthcheck": dial tcp 10.133.0.48:8943: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-6fcf89d9cd-bpq8m Readiness probe failed: Get "https://10.133.0.48:8444/healthz": dial tcp 10.133.0.48:8444: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-6fcf89d9cd-rbcnb Readiness probe errored and resulted in unknown state: rpc error: code = Canceled desc = context canceled 115m Warning Unhealthy pod/rhods-dashboard-6fcf89d9cd-rbcnb Readiness probe failed: Get "https://10.134.0.47:8943/healthcheck": dial tcp 10.134.0.47:8943: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-6fcf89d9cd-rbcnb Readiness probe failed: Get "https://10.134.0.47:8444/healthz": dial tcp 10.134.0.47:8444: connect: connection refused 81m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 81m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 22m Warning Unhealthy pod/mlflow-77b77b7789-qrvsw Readiness probe failed: Get "https://10.133.0.116:8443/mlflow/api/3.0/mlflow/server-info": dial tcp 10.133.0.116:8443: connect: connection refused 16m Warning Unhealthy pod/mlflow-8995d7b65-vtgb4 Readiness probe failed: Get "https://10.133.0.119:8443/mlflow/api/3.0/mlflow/server-info": dial tcp 10.133.0.119:8443: connect: connection refused 7m50s Warning Unhealthy pod/dashboard-operator-64d7c8d8db-qm6w6 Readiness probe failed: Get "http://10.132.0.51:8081/readyz": dial tcp 10.132.0.51:8081: connect: connection refused 7m4s Warning NodeNotReady pod/odh-notebook-controller-manager-777b5c4fdc-gcf4g Node is not ready 7m4s Warning NodeNotReady pod/kserve-localmodelnode-agent-fr6xv Node is not ready 7m4s Warning NodeNotReady pod/gen-ai-ui-6dcbf544f6-gtljh Node is not ready 7m4s Warning NodeNotReady pod/kserve-localmodel-controller-manager-5bc445646-fszr7 Node is not ready 7m4s Warning NodeNotReady pod/maas-ui-d874967cc-k49d8 Node is not ready 7m4s Warning NodeNotReady pod/kserve-module-controller-manager-775f5b5bc5-kdgd8 Node is not ready 7m4s Warning NodeNotReady pod/mlflow-operator-controller-manager-fbd674d66-f8fxp Node is not ready 7m4s Warning NodeNotReady pod/autorag-ui-76b845c4d4-7rqdl Node is not ready 7m4s Warning NodeNotReady pod/kserve-localmodelnode-agent-vfxwm Node is not ready 7m4s Warning NodeNotReady pod/ogx-k8s-operator-controller-manager-58cdd76679-n58wz Node is not ready 7m4s Warning NodeNotReady pod/spark-operator-webhook-7f794cfbb7-sk7fg Node is not ready 7m4s Warning NodeNotReady pod/notebooks-ui-5c6747cf85-gjzm7 Node is not ready 7m4s Warning NodeNotReady pod/catalog-controller-manager-5dcddfb9cc-hb8wk Node is not ready 7m4s Warning NodeNotReady pod/mlflow-ui-9758df657-4sjln Node is not ready 7m3s Warning NodeNotReady pod/trainer-operator-controller-manager-5dd7dfcb56-s569b Node is not ready 7m3s Warning NodeNotReady pod/opendatahub-feast-operator-6d65f59c96-tpdhf Node is not ready 7m3s Warning NodeNotReady pod/kuberay-operator-668c5f59bd-nbscf Node is not ready 7m3s Warning NodeNotReady pod/opendatahub-ogx-operator-86649f78d7-ltpbg Node is not ready 7m3s Warning NodeNotReady pod/kubeflow-trainer-controller-manager-696cfc9d78-cqjdc Node is not ready 7m3s Warning NodeNotReady pod/odh-model-controller-849d64c4b4-t2bsl Node is not ready 7m3s Warning NodeNotReady pod/kserve-controller-manager-549db7d5ff-df684 Node is not ready 7m3s Warning NodeNotReady pod/agent-ops-ui-69d9689c9-48qms Node is not ready 7m3s Warning NodeNotReady pod/llmisvc-controller-manager-6cf9b4f89c-s8j76 Node is not ready 7m3s Warning NodeNotReady pod/dashboard-operator-64d7c8d8db-qm6w6 Node is not ready 7m3s Warning NodeNotReady pod/notebook-controller-deployment-db7d4bdbf-545g7 Node is not ready 7m3s Warning NodeNotReady pod/rhods-dashboard-86644c7f4d-qjlpz Node is not ready 7m3s Warning NodeNotReady pod/feast-operator-controller-manager-fcff4ddc6-nb2mr Node is not ready 7m3s Warning NodeNotReady pod/model-registry-ui-85798dff55-fxb57 Node is not ready 7m3s Warning NodeNotReady pod/dashboard-redirect-576b457675-hfd6j Node is not ready 7m3s Warning NodeNotReady pod/eval-hub-ui-55bcbd4779-qdtf5 Node is not ready 7m3s Warning NodeNotReady pod/data-science-pipelines-operator-controller-manager-7847455bnmgk Node is not ready 7m3s Warning NodeNotReady pod/model-registry-operator-controller-manager-68589b648b-5bh5j Node is not ready 7m3s Warning NodeNotReady pod/trustyai-service-operator-controller-manager-8dddd65c6-rz5zx Node is not ready 7m3s Warning NodeNotReady pod/aihub-controller-manager-85bff87dd6-znlnr Node is not ready 7m3s Warning NodeNotReady pod/rhods-dashboard-86644c7f4d-t9w6n Node is not ready 7m3s Warning NodeNotReady pod/workbenches-operator-5cb6c594f8-t7dz4 Node is not ready 7m3s Warning NodeNotReady pod/ai-gateway-operator-74b6dcfcdc-7pzlr Node is not ready 7m2s Warning NodeNotReady pod/dashboard-redirect-576b457675-5b4z2 Node is not ready 7m2s Warning NodeNotReady pod/automl-ui-78945f54f8-vmbs5 Node is not ready 7m2s Warning NodeNotReady pod/spark-operator-module-controller-manager-565f8d557c-l67wm Node is not ready 7m2s Warning NodeNotReady pod/spark-operator-controller-7558766b8d-rl2hr Node is not ready 7m2s Warning NodeNotReady pod/maas-controller-7ddc487d54-n6kd7 Node is not ready 7m2s Warning NodeNotReady pod/odh-observability-dbfff9446-t6d6x Node is not ready 7m2s Warning NodeNotReady pod/model-serving-api-8445b5bcb7-d548m Node is not ready 5m46s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: skipping Pod rhods-dashboard-86644c7f4d-t9w6n for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-136-8.ec2.internal Not Found 61s Warning FailedCreate replicaset/aihub-controller-manager-85bff87dd6 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 61s Warning FailedCreate replicaset/spark-operator-webhook-7f794cfbb7 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 60s Warning FailedCreate replicaset/opendatahub-ogx-operator-86649f78d7 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 59s Warning FailedCreate replicaset/notebook-controller-deployment-db7d4bdbf Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 59s Warning FailedCreate replicaset/odh-model-controller-849d64c4b4 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 56s Warning FailedCreate replicaset/automl-ui-78945f54f8 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/maas-ui-d874967cc Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 45s Warning FailedCreate replicaset/kuberay-operator-668c5f59bd Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 44s Warning FailedCreate replicaset/model-serving-api-8445b5bcb7 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 44s Warning FailedCreate replicaset/spark-operator-module-controller-manager-565f8d557c Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 43s Warning FailedCreate replicaset/odh-observability-dbfff9446 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 43s Warning FailedCreate replicaset/opendatahub-feast-operator-6d65f59c96 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 43s Warning FailedCreate replicaset/model-registry-ui-85798dff55 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 8s Warning FailedCreate replicaset/dashboard-redirect-576b457675 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- LAST SEEN TYPE REASON OBJECT MESSAGE 4s Warning FailedScheduling pod/rhods-operator-6d7db7fb99-qhrvv no nodes available to schedule pods 5m53s Warning FailedScheduling pod/rhods-operator-6d7db7fb99-bl9kl 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 38s Warning FailedScheduling pod/rhods-operator-6d7db7fb99-bl9kl no nodes available to schedule pods 34s Warning FailedScheduling pod/rhods-operator-6d7db7fb99-bl9kl no nodes available to schedule pods 5m31s Warning FailedScheduling pod/rhods-operator-6d7db7fb99-qhrvv 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 8s Warning FailedScheduling pod/rhods-operator-6d7db7fb99-qhrvv no nodes available to schedule pods 117m Warning AppliedWithWarnings installplan/install-xn5bt 1 warning(s) generated during installation of operator "rhods-operator.3.6.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 7m5s Warning NodeNotReady pod/rhods-operator-6d7db7fb99-fsxnj Node is not ready 7m4s Warning NodeNotReady pod/rhods-operator-6d7db7fb99-k9njp Node is not ready 7m3s Warning NodeNotReady pod/rhods-operator-6d7db7fb99-qbxmv Node is not ready 7m3s Warning ComponentUnhealthy clusterserviceversion/rhods-operator.3.6.0-ea.2 installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. 5m45s Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: skipping Pod rhods-operator-6d7db7fb99-fsxnj for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-136-8.ec2.internal Not Found 5m45s Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: skipping Pod rhods-operator-6d7db7fb99-fsxnj for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-136-8.ec2.internal Not Found 119s Warning InstallCheckFailed clusterserviceversion/rhods-operator.3.6.0-ea.2 install timeout 8s Warning FailedCreate replicaset/rhods-operator-6d7db7fb99 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found --- namespace/rhoai-model-registries --- LAST SEEN TYPE REASON OBJECT MESSAGE 8s Warning FailedScheduling pod/model-catalog-postgres-69664bc65d-gwp7p no nodes available to schedule pods 83m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 83m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 85m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 5m32s Warning FailedScheduling pod/model-catalog-postgres-69664bc65d-gwp7p 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 4s Warning FailedScheduling pod/model-catalog-postgres-69664bc65d-gwp7p no nodes available to schedule pods 4s Warning FailedScheduling pod/model-catalog-6c54df8856-j9ffq no nodes available to schedule pods 8s Warning FailedScheduling pod/model-catalog-6c54df8856-j9ffq no nodes available to schedule pods 5m33s Warning FailedScheduling pod/model-catalog-6c54df8856-j9ffq 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 116m Warning FailedMount pod/model-catalog-6c54df8856-m4dlg MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 116m Warning Unhealthy pod/model-catalog-6c54df8856-m4dlg Readiness probe failed: Get "http://10.132.0.61:8080/readyz": dial tcp 10.132.0.61:8080: connect: connection refused 85m Warning FailedMount pod/model-registry0-797fd97cf6-hwg7w MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-registry0-kube-rbac-proxy-config" not found 7m4s Warning NodeNotReady pod/model-catalog-postgres-69664bc65d-fv7x8 Node is not ready 7m4s Warning NodeNotReady pod/model-catalog-6c54df8856-m4dlg Node is not ready --- namespace/rhods-notebooks --- (no events) --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-09-15T08:53:24Z) (no matching error/webhook lines in operator logs for this time window) --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-6d7db7fb99-bl9kl.log ==================== ==================== OPERATOR POD LOG: rhods-operator-6d7db7fb99-qhrvv.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/agent-ops-ui-69d9689c9-92ftx.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/ai-gateway-operator-74b6dcfcdc-bck56.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/autorag-ui-76b845c4d4-hbds7.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/catalog-controller-manager-5dcddfb9cc-stxb6.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-operator-64d7c8d8db-49k5n.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-576b457675-sx4mh.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/data-science-pipelines-operator-controller-manager-7847455mw242.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/eval-hub-ui-55bcbd4779-tzkrh.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/feast-operator-controller-manager-fcff4ddc6-gc74x.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/gen-ai-ui-6dcbf544f6-5l748.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-controller-manager-549db7d5ff-g6kqx.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-localmodel-controller-manager-5bc445646-vfng7.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-module-controller-manager-775f5b5bc5-jjls9.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kubeflow-trainer-controller-manager-696cfc9d78-mc2bv.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/llmisvc-controller-manager-6cf9b4f89c-lfcsl.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-controller-7ddc487d54-pjc6b.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/mlflow-operator-controller-manager-fbd674d66-m88rh.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/mlflow-ui-9758df657-l8vmz.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-registry-operator-controller-manager-68589b648b-ffdht.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/notebooks-ui-5c6747cf85-6s2m9.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-notebook-controller-manager-777b5c4fdc-ptgrq.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/ogx-k8s-operator-controller-manager-58cdd76679-fxqkm.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/rhods-dashboard-86644c7f4d-hg8q7.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-controller-7558766b8d-tcbfb.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/trainer-operator-controller-manager-5dd7dfcb56-wbnx2.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/trustyai-service-operator-controller-manager-8dddd65c6-4zpg4.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/workbenches-operator-5cb6c594f8-kgj4p.log ==================== ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-6c54df8856-j9ffq.log ==================== ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-postgres-69664bc65d-gwp7p.log ====================