==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== redhat-ods-applications/agent-ops-ui: ready=0/1 available=0 redhat-ods-applications/ai-gateway-operator: ready=0/1 available=0 redhat-ods-applications/aihub-controller-manager: ready=0/1 available=0 redhat-ods-applications/automl-ui: ready=0/1 available=0 redhat-ods-applications/autorag-ui: ready=0/1 available=0 redhat-ods-applications/catalog-controller-manager: ready=0/1 available=0 redhat-ods-applications/dashboard-operator: ready=0/1 available=0 redhat-ods-applications/dashboard-redirect: ready=0/2 available=0 redhat-ods-applications/data-science-pipelines-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/eval-hub-ui: ready=0/1 available=0 redhat-ods-applications/feast-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/gen-ai-ui: ready=0/1 available=0 redhat-ods-applications/kserve-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-localmodel-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-trainer-controller-manager: ready=0/1 available=0 redhat-ods-applications/kuberay-operator: ready=0/1 available=0 redhat-ods-applications/llmisvc-controller-manager: ready=0/1 available=0 redhat-ods-applications/maas-controller: ready=0/1 available=0 redhat-ods-applications/maas-ui: ready=0/1 available=0 redhat-ods-applications/mlflow-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/mlflow-ui: ready=0/1 available=0 redhat-ods-applications/model-registry-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-registry-ui: ready=0/1 available=0 redhat-ods-applications/model-serving-api: ready=0/1 available=0 redhat-ods-applications/notebook-controller-deployment: ready=0/1 available=0 redhat-ods-applications/notebooks-ui: ready=0/1 available=0 redhat-ods-applications/odh-model-controller: ready=0/1 available=0 redhat-ods-applications/odh-notebook-controller-manager: ready=0/1 available=0 redhat-ods-applications/ogx-k8s-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/opendatahub-feast-operator: ready=0/1 available=0 redhat-ods-applications/opendatahub-ogx-operator: ready=0/1 available=0 redhat-ods-applications/postgres-deployment: ready=0/1 available=0 redhat-ods-applications/rhods-dashboard: ready=0/2 available=0 redhat-ods-applications/spark-operator-controller: ready=0/1 available=0 redhat-ods-applications/spark-operator-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook: ready=0/1 available=0 redhat-ods-applications/trainer-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/trustyai-service-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/workbenches-operator: ready=0/1 available=0 redhat-ods-operator/rhods-operator: ready=0/3 available=0 rhoai-model-registries/model-catalog: ready=0/1 available=0 rhoai-model-registries/model-catalog-postgres: ready=0/1 available=0 83m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-8vxj8 no nodes available to schedule pods 22m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-8vxj8 no nodes available to schedule pods 6m24s Warning FailedScheduling pod/agent-ops-ui-77d6cbf9b6-7949c 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/agent-ops-ui-77d6cbf9b6-7949c no nodes available to schedule pods 55s Warning FailedScheduling pod/agent-ops-ui-77d6cbf9b6-7949c no nodes available to schedule pods 6m25s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-8vxj8 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 8m41s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 32m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m25s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-7v42f 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-7v42f no nodes available to schedule pods 55s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-7v42f no nodes available to schedule pods 6m24s Warning FailedScheduling pod/kuberay-operator-7d87f54798-sj576 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/kuberay-operator-7d87f54798-sj576 no nodes available to schedule pods 6m25s Warning FailedScheduling pod/notebooks-ui-6577cc467-rww4w 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 55s Warning FailedScheduling pod/kuberay-operator-7d87f54798-sj576 no nodes available to schedule pods 32m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/workbenches-operator-c674d9c9-r8gwc no nodes available to schedule pods 60s Warning FailedScheduling pod/workbenches-operator-c674d9c9-r8gwc no nodes available to schedule pods 6m26s Warning FailedScheduling pod/workbenches-operator-c674d9c9-r8gwc 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m3s Warning FailedScheduling pod/postgres-deployment-86754cb759-kz4l6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/notebooks-ui-6577cc467-rww4w no nodes available to schedule pods 55s Warning FailedScheduling pod/notebooks-ui-6577cc467-rww4w no nodes available to schedule pods 34m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 42m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 58m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 8m41s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 42m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 58m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 42m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 8m41s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 58m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/trustyai-service-operator-controller-manager-d6bf47757-fmvk7 no nodes available to schedule pods 60s Warning FailedScheduling pod/trustyai-service-operator-controller-manager-d6bf47757-fmvk7 no nodes available to schedule pods 6m26s Warning FailedScheduling pod/trustyai-service-operator-controller-manager-d6bf47757-fmvk7 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m26s Warning FailedScheduling pod/autorag-ui-58f548cbd-jj2mn 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/autorag-ui-58f548cbd-jj2mn no nodes available to schedule pods 55s Warning FailedScheduling pod/autorag-ui-58f548cbd-jj2mn no nodes available to schedule pods 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m26s Warning FailedScheduling pod/maas-controller-56c898799-86pd2 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60s Warning FailedScheduling pod/maas-controller-56c898799-86pd2 no nodes available to schedule pods 55s Warning FailedScheduling pod/opendatahub-ogx-operator-644d45d5b-pnx45 no nodes available to schedule pods 60s Warning FailedScheduling pod/opendatahub-ogx-operator-644d45d5b-pnx45 no nodes available to schedule pods 6m25s Warning FailedScheduling pod/opendatahub-ogx-operator-644d45d5b-pnx45 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 55s Warning FailedScheduling pod/maas-controller-56c898799-86pd2 no nodes available to schedule pods 6m3s Warning FailedScheduling pod/spark-operator-controller-c594b967f-7w6tc 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/spark-operator-controller-c594b967f-7w6tc no nodes available to schedule pods 55s Warning FailedScheduling pod/spark-operator-controller-c594b967f-7w6tc no nodes available to schedule pods 55s Warning FailedScheduling pod/model-registry-operator-controller-manager-6cbf9bb5c6-t2lsh no nodes available to schedule pods 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 58m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 42m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 8m41s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60s Warning FailedScheduling pod/model-registry-operator-controller-manager-6cbf9bb5c6-t2lsh no nodes available to schedule pods 6m4s Warning FailedScheduling pod/model-registry-operator-controller-manager-6cbf9bb5c6-t2lsh 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 83m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-k48b4 no nodes available to schedule pods 6m25s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-k48b4 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m3s Warning FailedScheduling pod/dashboard-redirect-7599b555cb-rtg8q 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/dashboard-redirect-7599b555cb-rtg8q no nodes available to schedule pods 55s Warning FailedScheduling pod/dashboard-redirect-7599b555cb-rtg8q no nodes available to schedule pods 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/mlflow-ui-7bb6d46f5f-kmz4t no nodes available to schedule pods 60s Warning FailedScheduling pod/mlflow-ui-7bb6d46f5f-kmz4t no nodes available to schedule pods 6m25s Warning FailedScheduling pod/mlflow-ui-7bb6d46f5f-kmz4t 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 70m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m25s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f894qp78 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f894qp78 no nodes available to schedule pods 55s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f894qp78 no nodes available to schedule pods 59m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-k48b4 no nodes available to schedule pods 59m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 58m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 42m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m25s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-ft4xm 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-ft4xm no nodes available to schedule pods 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m26s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-jrzp6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m26s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-qjbnd 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-jrzp6 no nodes available to schedule pods 55s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-jrzp6 no nodes available to schedule pods 60s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-qjbnd no nodes available to schedule pods 55s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-ft4xm no nodes available to schedule pods 32m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 32m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/rhods-dashboard-59fdb98f4c-8psxb no nodes available to schedule pods 60s Warning FailedScheduling pod/rhods-dashboard-59fdb98f4c-8psxb no nodes available to schedule pods 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/maas-ui-6978cf897c-pjvbd no nodes available to schedule pods 6m26s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-vjh8b 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-vjh8b no nodes available to schedule pods 55s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-vjh8b no nodes available to schedule pods 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-qjbnd no nodes available to schedule pods 6m3s Warning FailedScheduling pod/rhods-dashboard-59fdb98f4c-8psxb 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60s Warning FailedScheduling pod/maas-ui-6978cf897c-pjvbd no nodes available to schedule pods 6m24s Warning FailedScheduling pod/maas-ui-6978cf897c-pjvbd 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 22m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 8m41s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/opendatahub-feast-operator-84ff599545-5g4c4 no nodes available to schedule pods 60s Warning FailedScheduling pod/opendatahub-feast-operator-84ff599545-5g4c4 no nodes available to schedule pods 6m25s Warning FailedScheduling pod/opendatahub-feast-operator-84ff599545-5g4c4 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m24s Warning FailedScheduling pod/gen-ai-ui-5f8988f89b-2xr2z 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/gen-ai-ui-5f8988f89b-2xr2z no nodes available to schedule pods 55s Warning FailedScheduling pod/gen-ai-ui-5f8988f89b-2xr2z no nodes available to schedule pods 13m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 117m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 117m Warning FailedMount pod/odh-observability-59db79cb46-4klcg MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 117m Warning FailedMount pod/kuberay-operator-7d87f54798-bsx9k MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 117m Warning FailedMount pod/catalog-controller-manager-5597d8c757-wrlb9 MountVolume.SetUp failed for volume "cert" : secret "catalog-webhook-service" not found 117m Warning FailedMount pod/rhods-dashboard-76b64bfc7-9ktmn MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 117m Warning FailedMount pod/rhods-dashboard-76b64bfc7-x968c MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 117m Warning FailedMount pod/model-registry-ui-7cf875d474-s4n2p MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 117m Warning FailedMount pod/automl-ui-74d9b97d49-n7z6x MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 117m Warning FailedMount pod/autorag-ui-58f548cbd-4vfqq MountVolume.SetUp failed for volume "proxy-tls" : secret "autorag-proxy-tls" not found 117m Warning FailedMount pod/notebooks-ui-6577cc467-7qlxx MountVolume.SetUp failed for volume "proxy-tls" : secret "notebooks-proxy-tls" not found 117m Warning FailedMount pod/gen-ai-ui-5f8988f89b-jz6m7 MountVolume.SetUp failed for volume "proxy-tls" : secret "gen-ai-proxy-tls" not found 117m Warning FailedMount pod/maas-ui-6978cf897c-4f5gh MountVolume.SetUp failed for volume "proxy-tls" : secret "maas-proxy-tls" not found 117m Warning FailedMount pod/odh-notebook-controller-manager-5b49759b5-8md4j MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 117m Warning FailedMount pod/agent-ops-ui-77d6cbf9b6-249b4 MountVolume.SetUp failed for volume "proxy-tls" : secret "agent-ops-proxy-tls" not found 117m Warning FailedMount pod/eval-hub-ui-869b89d9b4-d4ngf MountVolume.SetUp failed for volume "proxy-tls" : secret "eval-hub-proxy-tls" not found 116m Warning FailedMount pod/llmisvc-controller-manager-6df7b6c5d4-hwl4f MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 116m Warning FailedMount pod/odh-model-controller-6b9dd8b59d-dcg8f MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 116m Warning FailedMount pod/model-serving-api-7df8c9655f-7knxl MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-x968c Readiness probe failed: Get "https://10.134.0.42:8444/healthz": dial tcp 10.134.0.42:8444: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-x968c Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-x968c Readiness probe failed: Get "https://10.134.0.42:8943/healthcheck": dial tcp 10.134.0.42:8943: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-9ktmn Readiness probe failed: Get "https://10.133.0.47:8444/healthz": dial tcp 10.133.0.47:8444: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-9ktmn Readiness probe failed: Get "https://10.133.0.47:8943/healthcheck": dial tcp 10.133.0.47:8943: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-9ktmn Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-9ktmn Readiness probe failed: Get "https://10.133.0.47:8444/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 83m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 83m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 34m Warning FailedToRetrieveImagePullSecret pod/data-science-pipelines-operator-controller-manager-56b8f89gxdlr Unable to retrieve some image pull secrets (data-science-pipelines-operator-contro-d9fbdd90-dockercfg-q7zx9); attempting to pull the image may not succeed. 34m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-56b8f89gxdlr MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 33m Warning Unhealthy pod/rhods-dashboard-59fdb98f4c-qwgm4 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 33m Warning Unhealthy pod/rhods-dashboard-59fdb98f4c-qwgm4 Readiness probe failed: Get "https://10.134.0.45:8943/healthcheck": dial tcp 10.134.0.45:8943: connect: connection refused 33m Warning FailedMount pod/kuberay-operator-7d87f54798-74bxv MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 33m Warning Unhealthy pod/rhods-dashboard-59fdb98f4c-fq2lh Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 33m Warning Unhealthy pod/rhods-dashboard-59fdb98f4c-fq2lh Readiness probe failed: Get "https://10.133.0.52:8943/healthcheck": dial tcp 10.133.0.52:8943: connect: connection refused 33m Warning Unhealthy pod/rhods-dashboard-59fdb98f4c-qwgm4 Readiness probe failed: Get "https://10.134.0.45:8444/healthz": dial tcp 10.134.0.45:8444: connect: connection refused 33m Warning Unhealthy pod/rhods-dashboard-59fdb98f4c-fq2lh Readiness probe failed: Get "https://10.133.0.52:8444/healthz": dial tcp 10.133.0.52:8444: connect: connection refused 32m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 32m Warning FailedMount pod/odh-model-controller-57457c4c89-2zwr2 MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 32m Warning FailedMount pod/kserve-controller-manager-5fbbd6dd88-t5sk8 MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 32m Warning FailedMount pod/kserve-localmodel-controller-manager-fc8c879d7-czg45 MountVolume.SetUp failed for volume "cert" : secret "localmodel-webhook-server-cert" not found 32m Warning FailedMount pod/rhods-dashboard-76b64bfc7-bddbz MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 32m Warning FailedMount pod/rhods-dashboard-76b64bfc7-np98l MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 32m Warning Unhealthy pod/opendatahub-ogx-operator-644d45d5b-r5swr Readiness probe failed: Get "http://10.133.0.82:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 32m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-np98l Readiness probe failed: Get "https://10.133.0.92:8444/healthz": dial tcp 10.133.0.92:8444: connect: connection refused 32m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-np98l Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 32m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-np98l Readiness probe failed: Get "https://10.133.0.92:8943/healthcheck": dial tcp 10.133.0.92:8943: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-bddbz Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 31m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-bddbz Readiness probe failed: Get "https://10.134.0.115:8943/healthcheck": dial tcp 10.134.0.115:8943: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-bddbz Readiness probe failed: Get "https://10.134.0.115:8444/healthz": dial tcp 10.134.0.115:8444: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-5f58cb6787-7v4vz Readiness probe failed: Get "https://10.133.0.97:8444/healthz": dial tcp 10.133.0.97:8444: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-5f58cb6787-7v4vz Readiness probe failed: Get "https://10.133.0.97:8943/healthcheck": dial tcp 10.133.0.97:8943: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-5f58cb6787-7v4vz Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 30m Warning Unhealthy pod/rhods-dashboard-5f58cb6787-w2p6j Readiness probe failed: Get "https://10.134.0.116:8943/healthcheck": dial tcp 10.134.0.116:8943: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-5f58cb6787-w2p6j Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 30m Warning Unhealthy pod/rhods-dashboard-5f58cb6787-w2p6j Readiness probe failed: Get "https://10.134.0.116:8444/healthz": dial tcp 10.134.0.116:8444: connect: connection refused 27m Warning Unhealthy pod/model-serving-api-7df8c9655f-xz7gl Readiness probe failed: Get "https://10.132.0.112:8443/readyz": dial tcp 10.132.0.112:8443: connect: connection refused 26m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-9g4j2 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 26m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-9g4j2 Readiness probe failed: Get "https://10.134.0.118:8943/healthcheck": dial tcp 10.134.0.118:8943: connect: connection refused 26m Warning Unhealthy pod/model-serving-api-7df8c9655f-t9rhq Readiness probe failed: Get "https://10.132.0.121:8443/readyz": dial tcp 10.132.0.121:8443: connect: connection refused 26m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-nmfls Readiness probe failed: Get "https://10.133.0.95:8943/healthcheck": dial tcp 10.133.0.95:8943: connect: connection refused 26m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-nmfls Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 26m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-9g4j2 Readiness probe failed: Get "https://10.134.0.118:8444/healthz": dial tcp 10.134.0.118:8444: connect: connection refused 26m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-nmfls Readiness probe failed: Get "https://10.133.0.95:8444/healthz": dial tcp 10.133.0.95:8444: connect: connection refused 26m Warning FailedToRetrieveImagePullSecret pod/rhods-dashboard-54b9bdc4c4-v2tmn Unable to retrieve some image pull secrets (rhods-dashboard-dockercfg-8bxlm); attempting to pull the image may not succeed. 26m Warning FailedMount pod/aihub-controller-manager-d877888b8-tskvd MountVolume.SetUp failed for volume "metrics-certs" : secret "aihub-controller-manager-metrics-tls" not found 25m Warning FailedMount pod/rhods-dashboard-54b9bdc4c4-v2tmn MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-09-06T11:47:55Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.21.30 Kubernetes Version: v1.34.9 --- Namespaces --- namespace/opendatahub namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.2 Authorino Operator 1.4.2 authorino-operator.v1.4.1 Installing cert-manager-operator.v1.20.0 cert-manager Operator for Red Hat OpenShift 1.20.0 cert-manager-operator.v1.19.0 Installing custom-metrics-autoscaler.v2.19.0-3 Custom Metrics Autoscaler 2.19.0-3 custom-metrics-autoscaler.v2.19.0-2 Installing dns-operator.v1.4.1 DNS Operator 1.4.1 dns-operator.v1.4.0 Installing kueue-operator.v1.4.1 Red Hat build of Kueue 1.4.1 Installing limitador-operator.v1.4.1 Limitador Operator 1.4.1 limitador-operator.v1.4.0 Installing rhcl-operator.v1.4.2 Red Hat Connectivity Link 1.4.2 rhcl-operator.v1.4.1 Installing rhods-operator.3.6.0-ea.1 Red Hat OpenShift AI 3.6.0-ea.1 Installing --- CSVs not in Succeeded phase --- authorino-operator.v1.4.2: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace cert-manager-operator.v1.20.0: phase=Installing reason=Copied message=The operator is running in cert-manager-operator but is managing this namespace custom-metrics-autoscaler.v2.19.0-3: phase=Installing reason=Copied message=The operator is running in openshift-keda but is managing this namespace dns-operator.v1.4.1: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace kueue-operator.v1.4.1: phase=Installing reason=Copied message=The operator is running in openshift-kueue-operator but is managing this namespace limitador-operator.v1.4.1: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace rhcl-operator.v1.4.2: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace rhods-operator.3.6.0-ea.1: phase=Installing reason=InstallWaiting message=installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aigateway: managementState=Managed aipipelines: managementState=Managed batchGateway: managementState=Removed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Removed llamastackoperator: managementState=n/a mcplifecycleoperator: managementState=Removed mlflowoperator: managementState=Managed modelregistry: managementState=Managed modelsAsAService: managementState=Managed ogx: managementState=Managed ray: managementState=Managed sparkoperator: managementState=Managed trainer: managementState=Managed trainingoperator: managementState=Removed trustyai: managementState=Managed workbenches: managementState=Managed workbenchesV2: managementState=Removed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= ModulesReady: status=True reason= message= AIGatewayReady: status=True reason= message= AIPipelinesReady: status=True reason= message= BatchGatewayReady: status=False reason=Removed message=Submodule ManagementState is set to Removed DashboardReady: status=True reason=AllDependentsHealthy message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=False reason=Removed message=Component ManagementState is set to Removed MCPLifecycleOperatorReady: status=False reason=Removed message=Module ManagementState is set to Removed MLflowOperatorReady: status=True reason=Ready message=MLflowOperator is ready to manage MLflow custom resources ModelRegistryReady: status=True reason=AllDependentsHealthy message= ModelsAsAServiceReady: status=True reason=Ready message=modelsAsAService is Managed and deployments are available OGXReady: status=True reason=AllDependentsHealthy message= RayReady: status=True reason= message= SparkOperatorReady: status=True reason=AllDependentsHealthy message= TrainerReady: status=True reason= message= TrustyAIReady: status=True reason= message= WorkbenchesReady: status=True reason=ReconcileSuccess message=Workbenches component is ready WorkbenchesV2Ready: status=False reason=Removed message=Submodule ManagementState is set to Removed --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 117m Ready 2026-09-06T09:49:59Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=False reason=Removed message=Monitoring is not enabled MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in Monitoring CR OpenTelemetryCollectorAvailable: status=False reason=MetricsAndTracesNotConfigured message=Metrics and traces are not configured in Monitoring CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in Monitoring CR NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR PersesAvailable: status=False reason=MetricsAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured ProvisioningSucceeded: status=True reason=Available message= Ready: status=True reason=Available message= --- Component CRs (Ready) --- NAME READY REASON VERSION aigateway.components.platform.opendatahub.io/default-aigateway True 1.26.2 NAME APPNAMESPACE AGE aihub.components.platform.opendatahub.io/default-aihub redhat-ods-applications 14m NAME PHASE READY REASON URL AGE dashboard.components.platform.opendatahub.io/default-dashboard Ready True AllDependentsHealthy https://rh-ai.apps.b7fa010369a995fedcd5.prod.konflux-ocp-ci.dev/ 16m NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON mlflowoperator.components.platform.opendatahub.io/default-mlflowoperator True Ready NAME READY REASON DISTRIBUTION ogx.components.platform.opendatahub.io/default-ogx True AllDependentsHealthy Red Hat OpenShift AI NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON sparkoperator.components.platform.opendatahub.io/default-sparkoperator True AllDependentsHealthy NAME READY REASON AGE trainer.components.platform.opendatahub.io/default-trainer True 19m NAME READY REASON trustyai.components.platform.opendatahub.io/default-trustyai True NAME READY REASON PHASE workbenches.components.platform.opendatahub.io/default-workbenches True ReconcileSuccess Ready --- Deployments not fully ready (ODH/RHOAI namespaces) --- redhat-ods-applications/agent-ops-ui: ready=0/1 available=0 redhat-ods-applications/ai-gateway-operator: ready=0/1 available=0 redhat-ods-applications/aihub-controller-manager: ready=0/1 available=0 redhat-ods-applications/automl-ui: ready=0/1 available=0 redhat-ods-applications/autorag-ui: ready=0/1 available=0 redhat-ods-applications/catalog-controller-manager: ready=0/1 available=0 redhat-ods-applications/dashboard-operator: ready=0/1 available=0 redhat-ods-applications/dashboard-redirect: ready=0/2 available=0 redhat-ods-applications/data-science-pipelines-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/eval-hub-ui: ready=0/1 available=0 redhat-ods-applications/feast-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/gen-ai-ui: ready=0/1 available=0 redhat-ods-applications/kserve-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-localmodel-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-trainer-controller-manager: ready=0/1 available=0 redhat-ods-applications/kuberay-operator: ready=0/1 available=0 redhat-ods-applications/llmisvc-controller-manager: ready=0/1 available=0 redhat-ods-applications/maas-controller: ready=0/1 available=0 redhat-ods-applications/maas-ui: ready=0/1 available=0 redhat-ods-applications/mlflow-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/mlflow-ui: ready=0/1 available=0 redhat-ods-applications/model-registry-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-registry-ui: ready=0/1 available=0 redhat-ods-applications/model-serving-api: ready=0/1 available=0 redhat-ods-applications/notebook-controller-deployment: ready=0/1 available=0 redhat-ods-applications/notebooks-ui: ready=0/1 available=0 redhat-ods-applications/odh-model-controller: ready=0/1 available=0 redhat-ods-applications/odh-notebook-controller-manager: ready=0/1 available=0 redhat-ods-applications/ogx-k8s-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/opendatahub-feast-operator: ready=0/1 available=0 redhat-ods-applications/opendatahub-ogx-operator: ready=0/1 available=0 redhat-ods-applications/postgres-deployment: ready=0/1 available=0 redhat-ods-applications/rhods-dashboard: ready=0/2 available=0 redhat-ods-applications/spark-operator-controller: ready=0/1 available=0 redhat-ods-applications/spark-operator-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook: ready=0/1 available=0 redhat-ods-applications/trainer-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/trustyai-service-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/workbenches-operator: ready=0/1 available=0 redhat-ods-operator/rhods-operator: ready=0/3 available=0 rhoai-model-registries/model-catalog: ready=0/1 available=0 rhoai-model-registries/model-catalog-postgres: ready=0/1 available=0 --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- redhat-ods-applications/agent-ops-ui-77d6cbf9b6-7949c: phase=Pending reason= redhat-ods-applications/ai-gateway-operator-89f69845b-7v42f: phase=Pending reason= redhat-ods-applications/autorag-ui-58f548cbd-jj2mn: phase=Pending reason= redhat-ods-applications/dashboard-redirect-7599b555cb-rtg8q: phase=Pending reason= redhat-ods-applications/data-science-pipelines-operator-controller-manager-56b8f894qp78: phase=Pending reason= redhat-ods-applications/feast-operator-controller-manager-5fc6bb8d67-vjh8b: phase=Pending reason= redhat-ods-applications/gen-ai-ui-5f8988f89b-2xr2z: phase=Pending reason= redhat-ods-applications/kserve-localmodel-controller-manager-fc8c879d7-k48b4: phase=Pending reason= redhat-ods-applications/kuberay-operator-7d87f54798-sj576: phase=Pending reason= redhat-ods-applications/maas-controller-56c898799-86pd2: phase=Pending reason= redhat-ods-applications/maas-ui-6978cf897c-pjvbd: phase=Pending reason= redhat-ods-applications/mlflow-ui-7bb6d46f5f-kmz4t: phase=Pending reason= redhat-ods-applications/model-registry-operator-controller-manager-6cbf9bb5c6-t2lsh: phase=Pending reason= redhat-ods-applications/notebook-controller-deployment-75c8689979-8vxj8: phase=Pending reason= redhat-ods-applications/notebooks-ui-6577cc467-rww4w: phase=Pending reason= redhat-ods-applications/odh-model-controller-6b9dd8b59d-jrzp6: phase=Pending reason= redhat-ods-applications/ogx-k8s-operator-controller-manager-59777cdbbb-ft4xm: phase=Pending reason= redhat-ods-applications/opendatahub-feast-operator-84ff599545-5g4c4: phase=Pending reason= redhat-ods-applications/opendatahub-ogx-operator-644d45d5b-pnx45: phase=Pending reason= redhat-ods-applications/rhods-dashboard-59fdb98f4c-8psxb: phase=Pending reason= redhat-ods-applications/spark-operator-controller-c594b967f-7w6tc: phase=Pending reason= redhat-ods-applications/spark-operator-module-controller-manager-59549b5854-qjbnd: phase=Pending reason= redhat-ods-applications/trustyai-service-operator-controller-manager-d6bf47757-fmvk7: phase=Pending reason= redhat-ods-applications/workbenches-operator-c674d9c9-r8gwc: phase=Pending reason= redhat-ods-operator/rhods-operator-56dfd54544-4c5fk: phase=Pending reason= redhat-ods-operator/rhods-operator-56dfd54544-9krgb: phase=Pending reason= rhoai-model-registries/model-catalog-7677467d7d-wkxzk: phase=Pending reason= rhoai-model-registries/model-catalog-postgres-56bcdb7d6b-shd22: phase=Pending reason= --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/opendatahub --- (no events) --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 83m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-8vxj8 no nodes available to schedule pods 22m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-8vxj8 no nodes available to schedule pods 6m24s Warning FailedScheduling pod/agent-ops-ui-77d6cbf9b6-7949c 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/agent-ops-ui-77d6cbf9b6-7949c no nodes available to schedule pods 55s Warning FailedScheduling pod/agent-ops-ui-77d6cbf9b6-7949c no nodes available to schedule pods 6m25s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-8vxj8 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 8m41s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 32m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m25s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-7v42f 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-7v42f no nodes available to schedule pods 55s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-7v42f no nodes available to schedule pods 6m24s Warning FailedScheduling pod/kuberay-operator-7d87f54798-sj576 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/kuberay-operator-7d87f54798-sj576 no nodes available to schedule pods 6m25s Warning FailedScheduling pod/notebooks-ui-6577cc467-rww4w 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 55s Warning FailedScheduling pod/kuberay-operator-7d87f54798-sj576 no nodes available to schedule pods 32m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/workbenches-operator-c674d9c9-r8gwc no nodes available to schedule pods 60s Warning FailedScheduling pod/workbenches-operator-c674d9c9-r8gwc no nodes available to schedule pods 6m26s Warning FailedScheduling pod/workbenches-operator-c674d9c9-r8gwc 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m3s Warning FailedScheduling pod/postgres-deployment-86754cb759-kz4l6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/notebooks-ui-6577cc467-rww4w no nodes available to schedule pods 55s Warning FailedScheduling pod/notebooks-ui-6577cc467-rww4w no nodes available to schedule pods 34m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 42m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 58m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 8m41s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 42m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 58m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 42m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 8m41s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 58m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/trustyai-service-operator-controller-manager-d6bf47757-fmvk7 no nodes available to schedule pods 60s Warning FailedScheduling pod/trustyai-service-operator-controller-manager-d6bf47757-fmvk7 no nodes available to schedule pods 6m26s Warning FailedScheduling pod/trustyai-service-operator-controller-manager-d6bf47757-fmvk7 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m26s Warning FailedScheduling pod/autorag-ui-58f548cbd-jj2mn 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/autorag-ui-58f548cbd-jj2mn no nodes available to schedule pods 55s Warning FailedScheduling pod/autorag-ui-58f548cbd-jj2mn no nodes available to schedule pods 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m26s Warning FailedScheduling pod/maas-controller-56c898799-86pd2 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60s Warning FailedScheduling pod/maas-controller-56c898799-86pd2 no nodes available to schedule pods 55s Warning FailedScheduling pod/opendatahub-ogx-operator-644d45d5b-pnx45 no nodes available to schedule pods 60s Warning FailedScheduling pod/opendatahub-ogx-operator-644d45d5b-pnx45 no nodes available to schedule pods 6m25s Warning FailedScheduling pod/opendatahub-ogx-operator-644d45d5b-pnx45 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 55s Warning FailedScheduling pod/maas-controller-56c898799-86pd2 no nodes available to schedule pods 6m3s Warning FailedScheduling pod/spark-operator-controller-c594b967f-7w6tc 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/spark-operator-controller-c594b967f-7w6tc no nodes available to schedule pods 55s Warning FailedScheduling pod/spark-operator-controller-c594b967f-7w6tc no nodes available to schedule pods 55s Warning FailedScheduling pod/model-registry-operator-controller-manager-6cbf9bb5c6-t2lsh no nodes available to schedule pods 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 59m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 58m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 42m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 8m41s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60s Warning FailedScheduling pod/model-registry-operator-controller-manager-6cbf9bb5c6-t2lsh no nodes available to schedule pods 6m4s Warning FailedScheduling pod/model-registry-operator-controller-manager-6cbf9bb5c6-t2lsh 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 83m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-k48b4 no nodes available to schedule pods 6m25s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-k48b4 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 100m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m3s Warning FailedScheduling pod/dashboard-redirect-7599b555cb-rtg8q 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/dashboard-redirect-7599b555cb-rtg8q no nodes available to schedule pods 55s Warning FailedScheduling pod/dashboard-redirect-7599b555cb-rtg8q no nodes available to schedule pods 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/mlflow-ui-7bb6d46f5f-kmz4t no nodes available to schedule pods 60s Warning FailedScheduling pod/mlflow-ui-7bb6d46f5f-kmz4t no nodes available to schedule pods 6m25s Warning FailedScheduling pod/mlflow-ui-7bb6d46f5f-kmz4t 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 70m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 64m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 62m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 61m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m25s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f894qp78 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f894qp78 no nodes available to schedule pods 55s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f894qp78 no nodes available to schedule pods 59m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-k48b4 no nodes available to schedule pods 59m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 58m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 42m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m25s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-ft4xm 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-ft4xm no nodes available to schedule pods 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m26s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-jrzp6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m26s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-qjbnd 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-jrzp6 no nodes available to schedule pods 55s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-jrzp6 no nodes available to schedule pods 60s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-qjbnd no nodes available to schedule pods 55s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-ft4xm no nodes available to schedule pods 32m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 32m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/rhods-dashboard-59fdb98f4c-8psxb no nodes available to schedule pods 60s Warning FailedScheduling pod/rhods-dashboard-59fdb98f4c-8psxb no nodes available to schedule pods 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/maas-ui-6978cf897c-pjvbd no nodes available to schedule pods 6m26s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-vjh8b 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-vjh8b no nodes available to schedule pods 55s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-vjh8b no nodes available to schedule pods 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-qjbnd no nodes available to schedule pods 6m3s Warning FailedScheduling pod/rhods-dashboard-59fdb98f4c-8psxb 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60s Warning FailedScheduling pod/maas-ui-6978cf897c-pjvbd no nodes available to schedule pods 6m24s Warning FailedScheduling pod/maas-ui-6978cf897c-pjvbd 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 22m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 8m41s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 55s Warning FailedScheduling pod/opendatahub-feast-operator-84ff599545-5g4c4 no nodes available to schedule pods 60s Warning FailedScheduling pod/opendatahub-feast-operator-84ff599545-5g4c4 no nodes available to schedule pods 6m25s Warning FailedScheduling pod/opendatahub-feast-operator-84ff599545-5g4c4 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m24s Warning FailedScheduling pod/gen-ai-ui-5f8988f89b-2xr2z 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60s Warning FailedScheduling pod/gen-ai-ui-5f8988f89b-2xr2z no nodes available to schedule pods 55s Warning FailedScheduling pod/gen-ai-ui-5f8988f89b-2xr2z no nodes available to schedule pods 13m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 117m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 117m Warning FailedMount pod/odh-observability-59db79cb46-4klcg MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 117m Warning FailedMount pod/kuberay-operator-7d87f54798-bsx9k MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 117m Warning FailedMount pod/catalog-controller-manager-5597d8c757-wrlb9 MountVolume.SetUp failed for volume "cert" : secret "catalog-webhook-service" not found 117m Warning FailedMount pod/rhods-dashboard-76b64bfc7-9ktmn MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 117m Warning FailedMount pod/rhods-dashboard-76b64bfc7-x968c MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 117m Warning FailedMount pod/model-registry-ui-7cf875d474-s4n2p MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 117m Warning FailedMount pod/automl-ui-74d9b97d49-n7z6x MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 117m Warning FailedMount pod/autorag-ui-58f548cbd-4vfqq MountVolume.SetUp failed for volume "proxy-tls" : secret "autorag-proxy-tls" not found 117m Warning FailedMount pod/notebooks-ui-6577cc467-7qlxx MountVolume.SetUp failed for volume "proxy-tls" : secret "notebooks-proxy-tls" not found 117m Warning FailedMount pod/gen-ai-ui-5f8988f89b-jz6m7 MountVolume.SetUp failed for volume "proxy-tls" : secret "gen-ai-proxy-tls" not found 117m Warning FailedMount pod/maas-ui-6978cf897c-4f5gh MountVolume.SetUp failed for volume "proxy-tls" : secret "maas-proxy-tls" not found 117m Warning FailedMount pod/odh-notebook-controller-manager-5b49759b5-8md4j MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 117m Warning FailedMount pod/agent-ops-ui-77d6cbf9b6-249b4 MountVolume.SetUp failed for volume "proxy-tls" : secret "agent-ops-proxy-tls" not found 117m Warning FailedMount pod/eval-hub-ui-869b89d9b4-d4ngf MountVolume.SetUp failed for volume "proxy-tls" : secret "eval-hub-proxy-tls" not found 116m Warning FailedMount pod/llmisvc-controller-manager-6df7b6c5d4-hwl4f MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 116m Warning FailedMount pod/odh-model-controller-6b9dd8b59d-dcg8f MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 116m Warning FailedMount pod/model-serving-api-7df8c9655f-7knxl MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-x968c Readiness probe failed: Get "https://10.134.0.42:8444/healthz": dial tcp 10.134.0.42:8444: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-x968c Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-x968c Readiness probe failed: Get "https://10.134.0.42:8943/healthcheck": dial tcp 10.134.0.42:8943: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-9ktmn Readiness probe failed: Get "https://10.133.0.47:8444/healthz": dial tcp 10.133.0.47:8444: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-9ktmn Readiness probe failed: Get "https://10.133.0.47:8943/healthcheck": dial tcp 10.133.0.47:8943: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-9ktmn Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 115m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-9ktmn Readiness probe failed: Get "https://10.133.0.47:8444/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 83m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 83m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 34m Warning FailedToRetrieveImagePullSecret pod/data-science-pipelines-operator-controller-manager-56b8f89gxdlr Unable to retrieve some image pull secrets (data-science-pipelines-operator-contro-d9fbdd90-dockercfg-q7zx9); attempting to pull the image may not succeed. 34m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-56b8f89gxdlr MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 33m Warning Unhealthy pod/rhods-dashboard-59fdb98f4c-qwgm4 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 33m Warning Unhealthy pod/rhods-dashboard-59fdb98f4c-qwgm4 Readiness probe failed: Get "https://10.134.0.45:8943/healthcheck": dial tcp 10.134.0.45:8943: connect: connection refused 33m Warning FailedMount pod/kuberay-operator-7d87f54798-74bxv MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 33m Warning Unhealthy pod/rhods-dashboard-59fdb98f4c-fq2lh Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 33m Warning Unhealthy pod/rhods-dashboard-59fdb98f4c-fq2lh Readiness probe failed: Get "https://10.133.0.52:8943/healthcheck": dial tcp 10.133.0.52:8943: connect: connection refused 33m Warning Unhealthy pod/rhods-dashboard-59fdb98f4c-qwgm4 Readiness probe failed: Get "https://10.134.0.45:8444/healthz": dial tcp 10.134.0.45:8444: connect: connection refused 33m Warning Unhealthy pod/rhods-dashboard-59fdb98f4c-fq2lh Readiness probe failed: Get "https://10.133.0.52:8444/healthz": dial tcp 10.133.0.52:8444: connect: connection refused 32m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 32m Warning FailedMount pod/odh-model-controller-57457c4c89-2zwr2 MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 32m Warning FailedMount pod/kserve-controller-manager-5fbbd6dd88-t5sk8 MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 32m Warning FailedMount pod/kserve-localmodel-controller-manager-fc8c879d7-czg45 MountVolume.SetUp failed for volume "cert" : secret "localmodel-webhook-server-cert" not found 32m Warning FailedMount pod/rhods-dashboard-76b64bfc7-bddbz MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 32m Warning FailedMount pod/rhods-dashboard-76b64bfc7-np98l MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 32m Warning Unhealthy pod/opendatahub-ogx-operator-644d45d5b-r5swr Readiness probe failed: Get "http://10.133.0.82:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 32m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-np98l Readiness probe failed: Get "https://10.133.0.92:8444/healthz": dial tcp 10.133.0.92:8444: connect: connection refused 32m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-np98l Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 32m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-np98l Readiness probe failed: Get "https://10.133.0.92:8943/healthcheck": dial tcp 10.133.0.92:8943: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-bddbz Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 31m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-bddbz Readiness probe failed: Get "https://10.134.0.115:8943/healthcheck": dial tcp 10.134.0.115:8943: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-bddbz Readiness probe failed: Get "https://10.134.0.115:8444/healthz": dial tcp 10.134.0.115:8444: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-5f58cb6787-7v4vz Readiness probe failed: Get "https://10.133.0.97:8444/healthz": dial tcp 10.133.0.97:8444: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-5f58cb6787-7v4vz Readiness probe failed: Get "https://10.133.0.97:8943/healthcheck": dial tcp 10.133.0.97:8943: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-5f58cb6787-7v4vz Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 30m Warning Unhealthy pod/rhods-dashboard-5f58cb6787-w2p6j Readiness probe failed: Get "https://10.134.0.116:8943/healthcheck": dial tcp 10.134.0.116:8943: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-5f58cb6787-w2p6j Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 30m Warning Unhealthy pod/rhods-dashboard-5f58cb6787-w2p6j Readiness probe failed: Get "https://10.134.0.116:8444/healthz": dial tcp 10.134.0.116:8444: connect: connection refused 27m Warning Unhealthy pod/model-serving-api-7df8c9655f-xz7gl Readiness probe failed: Get "https://10.132.0.112:8443/readyz": dial tcp 10.132.0.112:8443: connect: connection refused 26m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-9g4j2 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 26m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-9g4j2 Readiness probe failed: Get "https://10.134.0.118:8943/healthcheck": dial tcp 10.134.0.118:8943: connect: connection refused 26m Warning Unhealthy pod/model-serving-api-7df8c9655f-t9rhq Readiness probe failed: Get "https://10.132.0.121:8443/readyz": dial tcp 10.132.0.121:8443: connect: connection refused 26m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-nmfls Readiness probe failed: Get "https://10.133.0.95:8943/healthcheck": dial tcp 10.133.0.95:8943: connect: connection refused 26m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-nmfls Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 26m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-9g4j2 Readiness probe failed: Get "https://10.134.0.118:8444/healthz": dial tcp 10.134.0.118:8444: connect: connection refused 26m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-nmfls Readiness probe failed: Get "https://10.133.0.95:8444/healthz": dial tcp 10.133.0.95:8444: connect: connection refused 26m Warning FailedToRetrieveImagePullSecret pod/rhods-dashboard-54b9bdc4c4-v2tmn Unable to retrieve some image pull secrets (rhods-dashboard-dockercfg-8bxlm); attempting to pull the image may not succeed. 26m Warning FailedMount pod/aihub-controller-manager-d877888b8-tskvd MountVolume.SetUp failed for volume "metrics-certs" : secret "aihub-controller-manager-metrics-tls" not found 25m Warning FailedMount pod/rhods-dashboard-54b9bdc4c4-v2tmn MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 25m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-6pjvk Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 25m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-mksw5 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 25m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-mksw5 Readiness probe failed: Get "https://10.134.0.119:8943/healthcheck": dial tcp 10.134.0.119:8943: connect: connection refused 25m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-6pjvk Readiness probe failed: Get "https://10.133.0.102:8444/healthz": dial tcp 10.133.0.102:8444: connect: connection refused 25m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-6pjvk Readiness probe failed: Get "https://10.133.0.102:8943/healthcheck": dial tcp 10.133.0.102:8943: connect: connection refused 25m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-f2xkt Readiness probe failed: Get "https://10.133.0.103:8444/healthz": dial tcp 10.133.0.103:8444: connect: connection refused 25m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-f2xkt Readiness probe failed: Get "https://10.133.0.103:8943/healthcheck": dial tcp 10.133.0.103:8943: connect: connection refused 25m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-f2xkt Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 25m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-mksw5 Readiness probe failed: Get "https://10.134.0.119:8444/healthz": dial tcp 10.134.0.119:8444: connect: connection refused 25m Warning FailedMount pod/rhods-dashboard-54b9bdc4c4-v2tmn MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "kube-rbac-proxy-config" not found 25m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-v2tmn Readiness probe failed: Get "https://10.134.0.120:8444/healthz": dial tcp 10.134.0.120:8444: connect: connection refused 25m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-v2tmn Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 25m Warning Unhealthy pod/rhods-dashboard-54b9bdc4c4-v2tmn Readiness probe failed: Get "https://10.134.0.120:8943/healthcheck": dial tcp 10.134.0.120:8943: connect: connection refused 25m Warning FailedToUpdateEndpoint endpoints/aihub-controller-manager-metrics-service Failed to update endpoint redhat-ods-applications/aihub-controller-manager-metrics-service: Operation cannot be fulfilled on endpoints "aihub-controller-manager-metrics-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/aihub-controller-manager-metrics-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 27e35408-7b44-494f-85a2-4c7b73678c2d, UID in object meta: 24m Warning FailedMount pod/mlflow-operator-controller-manager-5bbb468f5f-5jhqq MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 23m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 23m Warning FailedMount pod/kserve-localmodel-controller-manager-fc8c879d7-bxc5z MountVolume.SetUp failed for volume "cert" : secret "localmodel-webhook-server-cert" not found 23m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 23m Warning FailedMount pod/odh-model-controller-57457c4c89-9zbjb MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 18m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 18m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 18m Warning FailedMount pod/odh-model-controller-57457c4c89-wf8kr MountVolume.SetUp failed for volume "metrics-certs" : secret "odh-model-controller-metrics-tls" not found 17m Warning FailedMount pod/odh-model-controller-57457c4c89-5q5js MountVolume.SetUp failed for volume "metrics-certs" : secret "odh-model-controller-metrics-tls" not found 17m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 17m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 17m Warning FailedMount pod/kserve-controller-manager-5fbbd6dd88-hj7pc MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 15m Warning FailedMount pod/mlflow-operator-controller-manager-5bbb468f5f-h66vg MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 15m Warning FailedMount pod/rhods-dashboard-76b64bfc7-24dpl MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 15m Warning FailedMount pod/rhods-dashboard-76b64bfc7-lbnzb MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 15m Warning FailedMount pod/maas-ui-6978cf897c-r8zh2 MountVolume.SetUp failed for volume "proxy-tls" : secret "maas-proxy-tls" not found 15m Warning FailedMount pod/mlflow-ui-7bb6d46f5f-m4sxs MountVolume.SetUp failed for volume "proxy-tls" : secret "mlflow-proxy-tls" not found 15m Warning FailedMount pod/agent-ops-ui-77d6cbf9b6-trb2k MountVolume.SetUp failed for volume "proxy-tls" : secret "agent-ops-proxy-tls" not found 14m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-24dpl Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 14m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-24dpl Readiness probe failed: Get "https://10.133.0.137:8943/healthcheck": dial tcp 10.133.0.137:8943: connect: connection refused 14m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-24dpl Readiness probe failed: Get "https://10.133.0.137:8444/healthz": dial tcp 10.133.0.137:8444: connect: connection refused 14m Warning FailedMount pod/automl-ui-74d9b97d49-9rqs2 MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 14m Warning FailedMount pod/autorag-ui-58f548cbd-fwgm9 MountVolume.SetUp failed for volume "proxy-tls" : secret "autorag-proxy-tls" not found 14m Warning FailedMount pod/trustyai-service-operator-controller-manager-d6bf47757-v4sdc MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 14m Warning FailedMount pod/eval-hub-ui-869b89d9b4-mgbp5 MountVolume.SetUp failed for volume "proxy-tls" : secret "eval-hub-proxy-tls" not found 13m Warning FailedMount pod/maas-controller-56c898799-bdgcz MountVolume.SetUp failed for volume "metrics-certs" : secret "maas-controller-metrics-tls" not found 13m Warning Unhealthy pod/rhods-dashboard-6b9f97b4f9-9bsl8 Readiness probe failed: Get "https://10.134.0.128:8444/healthz": dial tcp 10.134.0.128:8444: connect: connection refused 13m Warning Unhealthy pod/rhods-dashboard-6b9f97b4f9-9bsl8 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 13m Warning Unhealthy pod/rhods-dashboard-6b9f97b4f9-9bsl8 Readiness probe failed: Get "https://10.134.0.128:8943/healthcheck": dial tcp 10.134.0.128:8943: connect: connection refused 13m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-lbnzb Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 13m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-lbnzb Readiness probe failed: Get "https://10.134.0.127:8943/healthcheck": dial tcp 10.134.0.127:8943: connect: connection refused 13m Warning Unhealthy pod/rhods-dashboard-76b64bfc7-lbnzb Readiness probe failed: Get "https://10.134.0.127:8444/healthz": dial tcp 10.134.0.127:8444: connect: connection refused 12m Warning Unhealthy pod/rhods-dashboard-6b9f97b4f9-rcbqs Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 12m Warning Unhealthy pod/rhods-dashboard-6b9f97b4f9-rcbqs Readiness probe failed: Get "https://10.133.0.138:8943/healthcheck": dial tcp 10.133.0.138:8943: connect: connection refused 12m Warning FailedAttachVolume pod/mlflow-6cfdfd6c5d-l58cl Multi-Attach error for volume "pvc-a4bc2c97-e6c9-474c-b357-44c1bb00eb65" Volume is already exclusively attached to one node and can't be attached to another 12m Warning Unhealthy pod/rhods-dashboard-6b9f97b4f9-rcbqs Readiness probe failed: Get "https://10.133.0.138:8444/healthz": dial tcp 10.133.0.138:8444: connect: connection refused 8m15s Warning Unhealthy pod/mlflow-6cfdfd6c5d-l58cl Readiness probe failed: Get "https://10.134.0.133:8443/mlflow/api/3.0/mlflow/server-info": dial tcp 10.134.0.133:8443: connect: connection refused 8m4s Warning FailedAttachVolume pod/postgres-deployment-86754cb759-5sqbb AttachVolume.Attach failed for volume "pvc-e1c7e644-561b-43f8-9dcd-9d611e14e258" : rpc error: code = Internal desc = Could not attach volume "vol-05b2a680ae37563ef" to node "i-0edb7ce0609f85024": could not attach volume "vol-05b2a680ae37563ef" to node "i-0edb7ce0609f85024": operation error EC2: AttachVolume, https response error StatusCode: 400, RequestID: 369cd08f-4bbb-425f-a234-3d528942bdb1, api error IncorrectState: Instance 'i-0edb7ce0609f85024' is not 'running'. 8m3s Warning FailedAttachVolume pod/postgres-deployment-86754cb759-5sqbb AttachVolume.Attach failed for volume "pvc-e1c7e644-561b-43f8-9dcd-9d611e14e258" : rpc error: code = Internal desc = Could not attach volume "vol-05b2a680ae37563ef" to node "i-0edb7ce0609f85024": could not attach volume "vol-05b2a680ae37563ef" to node "i-0edb7ce0609f85024": operation error EC2: AttachVolume, https response error StatusCode: 400, RequestID: 2b4b6148-9735-4653-acb4-c74409b2800d, api error IncorrectState: Instance 'i-0edb7ce0609f85024' is not 'running'. 8m1s Warning FailedAttachVolume pod/postgres-deployment-86754cb759-5sqbb AttachVolume.Attach failed for volume "pvc-e1c7e644-561b-43f8-9dcd-9d611e14e258" : rpc error: code = Internal desc = Could not attach volume "vol-05b2a680ae37563ef" to node "i-0edb7ce0609f85024": could not attach volume "vol-05b2a680ae37563ef" to node "i-0edb7ce0609f85024": operation error EC2: AttachVolume, https response error StatusCode: 400, RequestID: 2616a55a-0ca8-4cf5-b47e-7597766a3f8b, api error IncorrectState: Instance 'i-0edb7ce0609f85024' is not 'running'. 7m54s Warning FailedAttachVolume pod/postgres-deployment-86754cb759-5sqbb AttachVolume.Attach failed for volume "pvc-e1c7e644-561b-43f8-9dcd-9d611e14e258" : rpc error: code = Internal desc = Could not attach volume "vol-05b2a680ae37563ef" to node "i-0edb7ce0609f85024": could not attach volume "vol-05b2a680ae37563ef" to node "i-0edb7ce0609f85024": operation error EC2: AttachVolume, https response error StatusCode: 400, RequestID: ca80bca9-4a73-48a6-84c5-a03ca3a97e43, api error IncorrectState: Instance 'i-0edb7ce0609f85024' is not 'running'. 7m45s Warning FailedAttachVolume pod/postgres-deployment-86754cb759-5sqbb AttachVolume.Attach failed for volume "pvc-e1c7e644-561b-43f8-9dcd-9d611e14e258" : rpc error: code = Internal desc = Could not attach volume "vol-05b2a680ae37563ef" to node "i-0edb7ce0609f85024": could not attach volume "vol-05b2a680ae37563ef" to node "i-0edb7ce0609f85024": operation error EC2: AttachVolume, https response error StatusCode: 400, RequestID: f52883d1-57a0-4602-a259-de501cdfc019, api error IncorrectState: Instance 'i-0edb7ce0609f85024' is not 'running'. 7m29s Warning FailedAttachVolume pod/postgres-deployment-86754cb759-5sqbb AttachVolume.Attach failed for volume "pvc-e1c7e644-561b-43f8-9dcd-9d611e14e258" : rpc error: code = Internal desc = Could not attach volume "vol-05b2a680ae37563ef" to node "i-0edb7ce0609f85024": could not attach volume "vol-05b2a680ae37563ef" to node "i-0edb7ce0609f85024": operation error EC2: AttachVolume, https response error StatusCode: 400, RequestID: 1d3867b8-cba2-4357-a1f2-ae0a9ed3d546, api error IncorrectState: Instance 'i-0edb7ce0609f85024' is not 'running'. 7m20s Warning NodeNotReady pod/feast-operator-controller-manager-5fc6bb8d67-nfqgr Node is not ready 7m20s Warning NodeNotReady pod/notebook-controller-deployment-75c8689979-wknbm Node is not ready 7m20s Warning NodeNotReady pod/model-registry-operator-controller-manager-6cbf9bb5c6-dcj4x Node is not ready 7m20s Warning NodeNotReady pod/spark-operator-controller-c594b967f-9lqp9 Node is not ready 7m20s Warning NodeNotReady pod/autorag-ui-58f548cbd-fwgm9 Node is not ready 7m20s Warning NodeNotReady pod/dashboard-redirect-7599b555cb-hjp42 Node is not ready 7m20s Warning NodeNotReady pod/rhods-dashboard-59fdb98f4c-kcc9f Node is not ready 7m20s Warning NodeNotReady pod/mlflow-ui-7bb6d46f5f-m4sxs Node is not ready 7m20s Warning NodeNotReady pod/kserve-localmodelnode-agent-p4nwc Node is not ready 7m20s Warning NodeNotReady pod/data-science-pipelines-operator-controller-manager-56b8f89l76cn Node is not ready 7m20s Warning NodeNotReady pod/kserve-localmodel-controller-manager-fc8c879d7-r5vfk Node is not ready 7m20s Warning NodeNotReady pod/agent-ops-ui-77d6cbf9b6-trb2k Node is not ready 7m20s Warning NodeNotReady pod/workbenches-operator-c674d9c9-tpnm4 Node is not ready 7m20s Warning NodeNotReady pod/opendatahub-feast-operator-84ff599545-dh2rk Node is not ready 7m20s Warning NodeNotReady pod/notebooks-ui-6577cc467-bt22n Node is not ready 7m19s Warning NodeNotReady pod/gen-ai-ui-5f8988f89b-5xx2v Node is not ready 7m19s Warning NodeNotReady pod/kuberay-operator-7d87f54798-hm66g Node is not ready 7m19s Warning NodeNotReady pod/odh-model-controller-6b9dd8b59d-svxxd Node is not ready 7m19s Warning NodeNotReady pod/ai-gateway-operator-89f69845b-k5pl8 Node is not ready 7m19s Warning NodeNotReady pod/maas-ui-6978cf897c-r8zh2 Node is not ready 7m19s Warning NodeNotReady pod/spark-operator-module-controller-manager-59549b5854-st6bh Node is not ready 7m19s Warning NodeNotReady pod/opendatahub-ogx-operator-644d45d5b-9qdjn Node is not ready 7m19s Warning NodeNotReady pod/ogx-k8s-operator-controller-manager-59777cdbbb-8wvvl Node is not ready 7m19s Warning NodeNotReady pod/trustyai-service-operator-controller-manager-d6bf47757-v4sdc Node is not ready 7m19s Warning NodeNotReady pod/maas-controller-56c898799-bdgcz Node is not ready 7m14s Warning NodeNotReady pod/model-registry-ui-7cf875d474-ms7ld Node is not ready 7m14s Warning NodeNotReady pod/catalog-controller-manager-5597d8c757-lsfjb Node is not ready 7m14s Warning NodeNotReady pod/aihub-controller-manager-d877888b8-nc99s Node is not ready 7m14s Warning NodeNotReady pod/model-serving-api-7df8c9655f-rxfnl Node is not ready 7m14s Warning NodeNotReady pod/kserve-module-controller-manager-66ff7c977-bnmwx Node is not ready 7m14s Warning NodeNotReady pod/mlflow-operator-controller-manager-5bbb468f5f-h66vg Node is not ready 7m14s Warning NodeNotReady pod/kserve-localmodelnode-agent-mh6lx Node is not ready 7m14s Warning NodeNotReady pod/dashboard-redirect-7599b555cb-4hjnn Node is not ready 7m13s Warning NodeNotReady pod/llmisvc-controller-manager-6df7b6c5d4-dgjjx Node is not ready 7m13s Warning NodeNotReady pod/kserve-controller-manager-5fbbd6dd88-hj7pc Node is not ready 7m13s Warning NodeNotReady pod/dashboard-operator-669d6564bd-2v27c Node is not ready 7m13s Warning NodeNotReady pod/automl-ui-74d9b97d49-9rqs2 Node is not ready 7m13s Warning NodeNotReady pod/eval-hub-ui-869b89d9b4-mgbp5 Node is not ready 7m13s Warning NodeNotReady pod/spark-operator-webhook-5d466b7d66-kf2p5 Node is not ready 7m13s Warning NodeNotReady pod/trainer-operator-controller-manager-6b585cbb9f-6lmqt Node is not ready 7m13s Warning NodeNotReady pod/rhods-dashboard-59fdb98f4c-wjfbx Node is not ready 7m13s Warning NodeNotReady pod/kubeflow-trainer-controller-manager-5449979c85-gmqxw Node is not ready 7m13s Warning NodeNotReady pod/odh-notebook-controller-manager-5b49759b5-p4f6k Node is not ready 6m56s Warning FailedAttachVolume pod/postgres-deployment-86754cb759-5sqbb AttachVolume.Attach failed for volume "pvc-e1c7e644-561b-43f8-9dcd-9d611e14e258" : csinode.storage.k8s.io "ip-10-0-128-233.ec2.internal" not found 47s Warning FailedCreate replicaset/aihub-controller-manager-d877888b8 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/llmisvc-controller-manager-6df7b6c5d4 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/postgres-deployment-86754cb759 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/model-registry-ui-7cf875d474 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/spark-operator-webhook-5d466b7d66 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/model-serving-api-7df8c9655f Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/dashboard-operator-669d6564bd Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/kserve-controller-manager-5fbbd6dd88 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/kubeflow-trainer-controller-manager-5449979c85 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/trainer-operator-controller-manager-6b585cbb9f Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/kserve-module-controller-manager-66ff7c977 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/dashboard-redirect-7599b555cb Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/catalog-controller-manager-5597d8c757 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/odh-notebook-controller-manager-5b49759b5 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/rhods-dashboard-59fdb98f4c Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/eval-hub-ui-869b89d9b4 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/automl-ui-74d9b97d49 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 47s Warning FailedCreate replicaset/mlflow-operator-controller-manager-5bbb468f5f Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- LAST SEEN TYPE REASON OBJECT MESSAGE 6m26s Warning FailedScheduling pod/rhods-operator-56dfd54544-4c5fk 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 61s Warning FailedScheduling pod/rhods-operator-56dfd54544-4c5fk no nodes available to schedule pods 56s Warning FailedScheduling pod/rhods-operator-56dfd54544-4c5fk no nodes available to schedule pods 56s Warning FailedScheduling pod/rhods-operator-56dfd54544-9krgb no nodes available to schedule pods 60s Warning FailedScheduling pod/rhods-operator-56dfd54544-9krgb no nodes available to schedule pods 6m4s Warning FailedScheduling pod/rhods-operator-56dfd54544-9krgb 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118m Warning AppliedWithWarnings installplan/install-4pxfz 1 warning(s) generated during installation of operator "rhods-operator.3.6.0-ea.1" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 7m21s Warning NodeNotReady pod/rhods-operator-56dfd54544-nvb6r Node is not ready 7m20s Warning ComponentUnhealthy clusterserviceversion/rhods-operator.3.6.0-ea.1 installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. 7m20s Warning NodeNotReady pod/rhods-operator-56dfd54544-n6rsl Node is not ready 7m15s Warning NodeNotReady pod/rhods-operator-56dfd54544-6dbvd Node is not ready 6m18s Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: skipping Pod rhods-operator-56dfd54544-n6rsl for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-128-233.ec2.internal Not Found 6m18s Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: skipping Pod rhods-operator-56dfd54544-n6rsl for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-128-233.ec2.internal Not Found 2m16s Warning InstallCheckFailed clusterserviceversion/rhods-operator.3.6.0-ea.1 install timeout 48s Warning FailedCreate replicaset/rhods-operator-56dfd54544 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found --- namespace/rhoai-model-registries --- LAST SEEN TYPE REASON OBJECT MESSAGE 87m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 87m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 6m26s Warning FailedScheduling pod/model-catalog-7677467d7d-wkxzk 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 61s Warning FailedScheduling pod/model-catalog-7677467d7d-wkxzk no nodes available to schedule pods 85m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 56s Warning FailedScheduling pod/model-catalog-7677467d7d-wkxzk no nodes available to schedule pods 56s Warning FailedScheduling pod/model-catalog-postgres-56bcdb7d6b-shd22 no nodes available to schedule pods 61s Warning FailedScheduling pod/model-catalog-postgres-56bcdb7d6b-shd22 no nodes available to schedule pods 6m25s Warning FailedScheduling pod/model-catalog-postgres-56bcdb7d6b-shd22 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 116m Warning FailedMount pod/model-catalog-68646fb9f4-4zc6n MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 87m Warning FailedMount pod/model-registry0-5bc586ffb6-5nh68 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-registry0-kube-rbac-proxy-config" not found 33m Warning Unhealthy pod/model-catalog-postgres-6774b8fd9d-dfjql Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 33m Warning Unhealthy pod/model-catalog-68646fb9f4-4zc6n Readiness probe failed: Get "http://10.134.0.46:8080/readyz": dial tcp 10.134.0.46:8080: connect: connection refused 33m Warning Unhealthy pod/model-catalog-68646fb9f4-4zc6n Readiness probe failed: Get "https://10.134.0.46:8888/healthz": dial tcp 10.134.0.46:8888: connect: connection refused 13m Warning FailedMount pod/model-catalog-7677467d7d-r9w9f MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 13m Warning Unhealthy pod/model-catalog-7677467d7d-r9w9f Readiness probe failed: Get "http://10.132.0.150:8080/readyz": dial tcp 10.132.0.150:8080: connect: connection refused 7m21s Warning NodeNotReady pod/model-catalog-7677467d7d-r9w9f Node is not ready 7m21s Warning NodeNotReady pod/model-catalog-postgres-56bcdb7d6b-454jm Node is not ready --- namespace/rhods-notebooks --- (no events) --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-09-06T09:19:59Z) (no matching error/webhook lines in operator logs for this time window) --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-56dfd54544-4c5fk.log ==================== ==================== OPERATOR POD LOG: rhods-operator-56dfd54544-9krgb.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/agent-ops-ui-77d6cbf9b6-7949c.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/ai-gateway-operator-89f69845b-7v42f.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/autorag-ui-58f548cbd-jj2mn.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-7599b555cb-rtg8q.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/data-science-pipelines-operator-controller-manager-56b8f894qp78.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/feast-operator-controller-manager-5fc6bb8d67-vjh8b.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/gen-ai-ui-5f8988f89b-2xr2z.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-localmodel-controller-manager-fc8c879d7-k48b4.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kuberay-operator-7d87f54798-sj576.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-controller-56c898799-86pd2.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-ui-6978cf897c-pjvbd.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/mlflow-ui-7bb6d46f5f-kmz4t.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-registry-operator-controller-manager-6cbf9bb5c6-t2lsh.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/notebook-controller-deployment-75c8689979-8vxj8.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/notebooks-ui-6577cc467-rww4w.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-model-controller-6b9dd8b59d-jrzp6.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/ogx-k8s-operator-controller-manager-59777cdbbb-ft4xm.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/opendatahub-feast-operator-84ff599545-5g4c4.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/opendatahub-ogx-operator-644d45d5b-pnx45.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/rhods-dashboard-59fdb98f4c-8psxb.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-controller-c594b967f-7w6tc.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-module-controller-manager-59549b5854-qjbnd.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/trustyai-service-operator-controller-manager-d6bf47757-fmvk7.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/workbenches-operator-c674d9c9-r8gwc.log ==================== ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-7677467d7d-wkxzk.log ==================== ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-postgres-56bcdb7d6b-shd22.log ====================