==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== redhat-ods-applications/agent-ops-ui: ready=0/1 available=0 redhat-ods-applications/ai-gateway-operator: ready=0/1 available=0 redhat-ods-applications/aihub-controller-manager: ready=0/1 available=0 redhat-ods-applications/automl-ui: ready=0/1 available=0 redhat-ods-applications/autorag-ui: ready=0/1 available=0 redhat-ods-applications/catalog-controller-manager: ready=0/1 available=0 redhat-ods-applications/dashboard-operator: ready=0/2 available=0 redhat-ods-applications/dashboard-redirect: ready=0/2 available=0 redhat-ods-applications/data-science-pipelines-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/eval-hub-ui: ready=0/1 available=0 redhat-ods-applications/feast-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/gen-ai-ui: ready=0/1 available=0 redhat-ods-applications/kserve-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-localmodel-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-trainer-controller-manager: ready=0/1 available=0 redhat-ods-applications/kuberay-operator: ready=0/1 available=0 redhat-ods-applications/llmisvc-controller-manager: ready=0/1 available=0 redhat-ods-applications/maas-controller: ready=0/1 available=0 redhat-ods-applications/maas-ui: ready=0/1 available=0 redhat-ods-applications/mlflow-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/mlflow-ui: ready=0/1 available=0 redhat-ods-applications/model-registry-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-registry-ui: ready=0/1 available=0 redhat-ods-applications/model-serving-api: ready=0/1 available=0 redhat-ods-applications/notebook-controller-deployment: ready=0/1 available=0 redhat-ods-applications/notebooks-ui: ready=0/1 available=0 redhat-ods-applications/odh-model-controller: ready=0/1 available=0 redhat-ods-applications/odh-notebook-controller-manager: ready=0/1 available=0 redhat-ods-applications/ogx-k8s-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/opendatahub-feast-operator: ready=0/1 available=0 redhat-ods-applications/opendatahub-ogx-operator: ready=0/1 available=0 redhat-ods-applications/postgres-deployment: ready=0/1 available=0 redhat-ods-applications/rhods-dashboard: ready=0/2 available=0 redhat-ods-applications/spark-operator-controller: ready=0/1 available=0 redhat-ods-applications/spark-operator-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook: ready=0/1 available=0 redhat-ods-applications/trainer-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/trustyai-service-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/workbenches-operator: ready=0/1 available=0 redhat-ods-operator/rhods-operator: ready=0/3 available=0 rhoai-model-registries/model-catalog: ready=0/1 available=0 rhoai-model-registries/model-catalog-postgres: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook-5d466b7d66-vnwd8: phase=Pending reason= 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m8s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m37s Warning FailedScheduling pod/agent-ops-ui-7df989bc55-wlhzf 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/agent-ops-ui-7df989bc55-wlhzf no nodes available to schedule pods 80s Warning FailedScheduling pod/agent-ops-ui-7df989bc55-wlhzf no nodes available to schedule pods 63m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 101m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 80s Warning FailedScheduling pod/model-registry-operator-controller-manager-6bcdfb5b87-hqrtn no nodes available to schedule pods 85s Warning FailedScheduling pod/model-registry-operator-controller-manager-6bcdfb5b87-hqrtn no nodes available to schedule pods 6m37s Warning FailedScheduling pod/model-registry-operator-controller-manager-6bcdfb5b87-hqrtn 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m59s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-w7969 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-w7969 no nodes available to schedule pods 110s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-w7969 no nodes available to schedule pods 110s Warning FailedScheduling pod/workbenches-operator-c674d9c9-62cwk no nodes available to schedule pods 115s Warning FailedScheduling pod/workbenches-operator-c674d9c9-62cwk no nodes available to schedule pods 6m59s Warning FailedScheduling pod/workbenches-operator-c674d9c9-62cwk 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m37s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-94b2h 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-94b2h no nodes available to schedule pods 80s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-94b2h no nodes available to schedule pods 6m37s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-6sgnz 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-6sgnz no nodes available to schedule pods 80s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-6sgnz no nodes available to schedule pods 6m38s Warning FailedScheduling pod/notebooks-ui-6577cc467-zfvzv 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/notebooks-ui-6577cc467-zfvzv no nodes available to schedule pods 80s Warning FailedScheduling pod/notebooks-ui-6577cc467-zfvzv no nodes available to schedule pods 105m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 101m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m8s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m58s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-bmf58 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-bmf58 no nodes available to schedule pods 110s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-bmf58 no nodes available to schedule pods 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m59s Warning FailedScheduling pod/automl-ui-74d9b97d49-fr8wz 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/automl-ui-74d9b97d49-fr8wz no nodes available to schedule pods 110s Warning FailedScheduling pod/automl-ui-74d9b97d49-fr8wz no nodes available to schedule pods 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 80s Warning FailedScheduling pod/maas-ui-675f94cfd8-l7g85 no nodes available to schedule pods 85s Warning FailedScheduling pod/maas-ui-675f94cfd8-l7g85 no nodes available to schedule pods 6m38s Warning FailedScheduling pod/maas-ui-675f94cfd8-l7g85 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 105m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m8s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 110s Warning FailedScheduling pod/spark-operator-webhook-5d466b7d66-vnwd8 no nodes available to schedule pods 115s Warning FailedScheduling pod/spark-operator-webhook-5d466b7d66-vnwd8 no nodes available to schedule pods 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m Warning FailedScheduling pod/spark-operator-webhook-5d466b7d66-vnwd8 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 16m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m59s Warning FailedScheduling pod/autorag-ui-58f548cbd-cm2s7 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/autorag-ui-58f548cbd-cm2s7 no nodes available to schedule pods 110s Warning FailedScheduling pod/autorag-ui-58f548cbd-cm2s7 no nodes available to schedule pods 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m38s Warning FailedScheduling pod/catalog-controller-manager-67d887945b-rkzjh 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/catalog-controller-manager-67d887945b-rkzjh no nodes available to schedule pods 80s Warning FailedScheduling pod/catalog-controller-manager-67d887945b-rkzjh no nodes available to schedule pods 110s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-hjqrv no nodes available to schedule pods 115s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-hjqrv no nodes available to schedule pods 6m59s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-hjqrv 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 52m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 101m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m8s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m37s Warning FailedScheduling pod/dashboard-operator-669d6564bd-c85d9 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/dashboard-operator-669d6564bd-c85d9 no nodes available to schedule pods 80s Warning FailedScheduling pod/dashboard-operator-669d6564bd-c85d9 no nodes available to schedule pods 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 101m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 110s Warning FailedScheduling pod/maas-controller-56c898799-rbr4l no nodes available to schedule pods 115s Warning FailedScheduling pod/maas-controller-56c898799-rbr4l no nodes available to schedule pods 6m58s Warning FailedScheduling pod/maas-controller-56c898799-rbr4l 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 101m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m37s Warning FailedScheduling pod/dashboard-redirect-55f4444684-xgtjk 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/dashboard-redirect-55f4444684-xgtjk no nodes available to schedule pods 80s Warning FailedScheduling pod/dashboard-redirect-55f4444684-xgtjk no nodes available to schedule pods 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 110s Warning FailedScheduling pod/spark-operator-controller-c594b967f-ds5f6 no nodes available to schedule pods 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 115s Warning FailedScheduling pod/spark-operator-controller-c594b967f-ds5f6 no nodes available to schedule pods 6m59s Warning FailedScheduling pod/spark-operator-controller-c594b967f-ds5f6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 80s Warning FailedScheduling pod/rhods-dashboard-bfcc58b46-x29br no nodes available to schedule pods 85s Warning FailedScheduling pod/rhods-dashboard-bfcc58b46-x29br no nodes available to schedule pods 6m37s Warning FailedScheduling pod/rhods-dashboard-bfcc58b46-x29br 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 110s Warning FailedScheduling pod/rhods-dashboard-bfcc58b46-cmzzz no nodes available to schedule pods 115s Warning FailedScheduling pod/rhods-dashboard-bfcc58b46-cmzzz no nodes available to schedule pods 6m59s Warning FailedScheduling pod/rhods-dashboard-bfcc58b46-cmzzz 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m58s Warning FailedScheduling pod/eval-hub-ui-869b89d9b4-wvcsk 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/eval-hub-ui-869b89d9b4-wvcsk no nodes available to schedule pods 110s Warning FailedScheduling pod/eval-hub-ui-869b89d9b4-wvcsk no nodes available to schedule pods 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m8s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m58s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-7rlmx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-7rlmx no nodes available to schedule pods 110s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-7rlmx no nodes available to schedule pods 80s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-zr9m6 no nodes available to schedule pods 85s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-zr9m6 no nodes available to schedule pods 6m38s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-zr9m6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m59s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-4lz7l 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-4lz7l no nodes available to schedule pods 110s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-4lz7l no nodes available to schedule pods 17m Warning FailedScheduling pod/kserve-localmodelnode-agent-zfds9 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 17m Warning FailedScheduling pod/kserve-localmodelnode-agent-vf2zp 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 19m Warning FailedScheduling pod/kserve-localmodelnode-agent-t6lz7 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 24m Warning FailedScheduling pod/kserve-localmodelnode-agent-n2qlk 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 24m Warning FailedScheduling pod/kserve-localmodelnode-agent-lwzt2 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 19m Warning FailedScheduling pod/kserve-localmodelnode-agent-8z62g 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 19m Warning FailedScheduling pod/kserve-localmodelnode-agent-8z62g 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 80s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-tsw29 no nodes available to schedule pods 85s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-tsw29 no nodes available to schedule pods 6m36s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-tsw29 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 119m Warning FailedMount pod/odh-observability-59db79cb46-jxh7c MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 119m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 119m Warning FailedMount pod/kuberay-operator-7d87f54798-z9q95 MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 119m Warning FailedMount pod/mlflow-operator-controller-manager-5bbb468f5f-4cnxp MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 119m Warning FailedMount pod/catalog-controller-manager-67d887945b-qlktz MountVolume.SetUp failed for volume "cert" : secret "catalog-webhook-service" not found 119m Warning FailedMount pod/model-registry-ui-67b8c946b9-s68np MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 119m Warning FailedMount pod/automl-ui-74d9b97d49-x268c MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 119m Warning FailedMount pod/mlflow-ui-7bb6d46f5f-nkqj6 MountVolume.SetUp failed for volume "proxy-tls" : secret "mlflow-proxy-tls" not found 118m Warning FailedMount pod/odh-notebook-controller-manager-5bcb7b86db-24424 MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 118m Warning FailedMount pod/trustyai-service-operator-controller-manager-d6bf47757-8djfr MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 117m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-b5ft6 Readiness probe failed: Get "https://10.134.0.38:8943/healthcheck": dial tcp 10.134.0.38:8943: connect: connection refused 117m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-b5ft6 Readiness probe failed: Get "https://10.134.0.38:8444/healthz": dial tcp 10.134.0.38:8444: connect: connection refused 117m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-b5ft6 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 117m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-2ncqt Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 117m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-2ncqt Readiness probe failed: Get "https://10.133.0.47:8943/healthcheck": dial tcp 10.133.0.47:8943: connect: connection refused 117m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-2ncqt Readiness probe failed: Get "https://10.133.0.47:8444/healthz": dial tcp 10.133.0.47:8444: connect: connection refused 101m Warning FailedMount pod/ogx-k8s-operator-controller-manager-59777cdbbb-x9blj MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 84m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 84m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 34m Warning FailedToRetrieveImagePullSecret pod/maas-controller-56c898799-t7zrv Unable to retrieve some image pull secrets (maas-controller-dockercfg-vp7c6); attempting to pull the image may not succeed. 34m Warning FailedMount pod/maas-controller-56c898799-t7zrv MountVolume.SetUp failed for volume "metrics-certs" : secret "maas-controller-metrics-tls" not found 34m Warning FailedMount pod/maas-controller-56c898799-t7zrv MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found 34m Warning Unhealthy pod/rhods-dashboard-bfcc58b46-v2xm8 Readiness probe failed: Get "https://10.132.0.55:8943/healthcheck": dial tcp 10.132.0.55:8943: connect: connection refused 34m Warning Unhealthy pod/kubeflow-trainer-controller-manager-5449979c85-h4tsl Readiness probe failed: Get "http://10.133.0.45:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning Unhealthy pod/rhods-dashboard-bfcc58b46-v2xm8 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 34m Warning FailedToUpdateEndpoint endpoints/aihub-controller-manager-metrics-service Failed to update endpoint redhat-ods-applications/aihub-controller-manager-metrics-service: Operation cannot be fulfilled on endpoints "aihub-controller-manager-metrics-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/aihub-controller-manager-metrics-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: a5481786-dca0-4fe8-b208-e35728ace0fb, UID in object meta: 34m Warning Unhealthy pod/rhods-dashboard-bfcc58b46-42vcs Readiness probe failed: Get "https://10.133.0.52:8943/healthcheck": dial tcp 10.133.0.52:8943: connect: connection refused 34m Warning Unhealthy pod/rhods-dashboard-bfcc58b46-42vcs Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 33m Warning Unhealthy pod/rhods-dashboard-bfcc58b46-42vcs Readiness probe failed: Get "https://10.133.0.52:8444/healthz": dial tcp 10.133.0.52:8444: connect: connection refused 33m Warning Unhealthy pod/rhods-dashboard-bfcc58b46-v2xm8 Readiness probe failed: Get "https://10.132.0.55:8444/healthz": dial tcp 10.132.0.55:8444: connect: connection refused 33m Warning FailedMount pod/kuberay-operator-7d87f54798-z4s96 MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 33m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 33m Warning FailedMount pod/odh-model-controller-57457c4c89-8j6h9 MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 33m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 33m Warning FailedMount pod/kserve-controller-manager-5fbbd6dd88-wrr9b MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 32m Warning FailedMount pod/automl-ui-74d9b97d49-lzkxj MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 32m Warning FailedMount pod/autorag-ui-58f548cbd-8pn4g MountVolume.SetUp failed for volume "proxy-tls" : secret "autorag-proxy-tls" not found 31m Warning Unhealthy pod/spark-operator-controller-c594b967f-wpdtn Readiness probe failed: Get "http://10.133.0.108:8081/readyz": dial tcp 10.133.0.108:8081: connect: connection refused 31m Warning Unhealthy pod/spark-operator-webhook-5d466b7d66-g8r7p Readiness probe failed: Get "http://10.133.0.109:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 31m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-zhmtt Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 31m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-zhmtt Readiness probe failed: Get "https://10.134.0.106:8943/healthcheck": dial tcp 10.134.0.106:8943: connect: connection refused 31m Warning FailedToUpdateEndpoint endpoints/odh-dashboard-autorag-ui Failed to update endpoint redhat-ods-applications/odh-dashboard-autorag-ui: Operation cannot be fulfilled on endpoints "odh-dashboard-autorag-ui": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/odh-dashboard-autorag-ui, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 79bb98aa-3dac-4c9d-ae1e-46c4f3573e85, UID in object meta: 31m Warning FailedToUpdateEndpoint endpoints/odh-dashboard-automl-ui Failed to update endpoint redhat-ods-applications/odh-dashboard-automl-ui: Operation cannot be fulfilled on endpoints "odh-dashboard-automl-ui": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/odh-dashboard-automl-ui, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 9931d5c4-97af-4cd3-9b95-ee5288a56884, UID in object meta: 31m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-zhmtt Readiness probe failed: Get "https://10.134.0.106:8444/healthz": dial tcp 10.134.0.106:8444: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-7f94fcf99d-rwzbj Readiness probe failed: Get "https://10.133.0.114:8444/healthz": dial tcp 10.133.0.114:8444: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-7f94fcf99d-rwzbj Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 31m Warning Unhealthy pod/rhods-dashboard-7f94fcf99d-rwzbj Readiness probe failed: Get "https://10.133.0.114:8943/healthcheck": dial tcp 10.133.0.114:8943: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-dgpl5 Readiness probe failed: Get "https://10.133.0.111:8943/healthcheck": dial tcp 10.133.0.111:8943: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-dgpl5 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 30m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-dgpl5 Readiness probe failed: Get "https://10.133.0.111:8444/healthz": dial tcp 10.133.0.111:8444: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-7f94fcf99d-6cjmj Readiness probe failed: Get "https://10.132.0.101:8943/healthcheck": dial tcp 10.132.0.101:8943: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-7f94fcf99d-6cjmj Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 29m Warning Unhealthy pod/rhods-dashboard-7f94fcf99d-6cjmj Readiness probe failed: Get "https://10.132.0.101:8444/healthz": dial tcp 10.132.0.101:8444: connect: connection refused 27m Warning Unhealthy pod/odh-model-controller-57457c4c89-qkzqq Readiness probe failed: Get "http://10.132.0.107:8081/readyz": dial tcp 10.132.0.107:8081: connect: connection refused 26m Warning FailedMount pod/model-registry-operator-controller-manager-69d5948bc6-z5tm6 MountVolume.SetUp failed for volume "server-cert" : secret "model-registry-operator-controller-manager-metrics-service" not found --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-09-07T11:51:33Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.21.30 Kubernetes Version: v1.34.9 --- Namespaces --- namespace/opendatahub namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.2 Authorino Operator 1.4.2 authorino-operator.v1.4.1 Installing cert-manager-operator.v1.20.0 cert-manager Operator for Red Hat OpenShift 1.20.0 cert-manager-operator.v1.19.0 Installing custom-metrics-autoscaler.v2.19.0-3 Custom Metrics Autoscaler 2.19.0-3 custom-metrics-autoscaler.v2.19.0-2 Installing dns-operator.v1.4.1 DNS Operator 1.4.1 dns-operator.v1.4.0 Installing kueue-operator.v1.4.1 Red Hat build of Kueue 1.4.1 Installing limitador-operator.v1.4.1 Limitador Operator 1.4.1 limitador-operator.v1.4.0 Installing rhcl-operator.v1.4.2 Red Hat Connectivity Link 1.4.2 rhcl-operator.v1.4.1 Installing rhods-operator.3.6.0-ea.1 Red Hat OpenShift AI 3.6.0-ea.1 Installing --- CSVs not in Succeeded phase --- authorino-operator.v1.4.2: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace cert-manager-operator.v1.20.0: phase=Installing reason=Copied message=The operator is running in cert-manager-operator but is managing this namespace custom-metrics-autoscaler.v2.19.0-3: phase=Installing reason=Copied message=The operator is running in openshift-keda but is managing this namespace dns-operator.v1.4.1: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace kueue-operator.v1.4.1: phase=Installing reason=Copied message=The operator is running in openshift-kueue-operator but is managing this namespace limitador-operator.v1.4.1: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace rhcl-operator.v1.4.2: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace rhods-operator.3.6.0-ea.1: phase=Installing reason=InstallWaiting message=installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aigateway: managementState=Managed aipipelines: managementState=Managed batchGateway: managementState=Removed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Removed llamastackoperator: managementState=n/a mcplifecycleoperator: managementState=Removed mlflowoperator: managementState=Managed modelregistry: managementState=Managed modelsAsAService: managementState=Managed ogx: managementState=Managed ray: managementState=Managed sparkoperator: managementState=Managed trainer: managementState=Managed trainingoperator: managementState=Removed trustyai: managementState=Managed workbenches: managementState=Managed workbenchesV2: managementState=Removed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= ModulesReady: status=True reason= message= AIGatewayReady: status=True reason= message= AIPipelinesReady: status=True reason= message= BatchGatewayReady: status=False reason=Removed message=Submodule ManagementState is set to Removed DashboardReady: status=True reason=AllDependentsHealthy message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=False reason=Removed message=Component ManagementState is set to Removed MCPLifecycleOperatorReady: status=False reason=Removed message=Module ManagementState is set to Removed MLflowOperatorReady: status=True reason=Ready message=MLflowOperator is ready to manage MLflow custom resources ModelRegistryReady: status=True reason=AllDependentsHealthy message= ModelsAsAServiceReady: status=True reason=Ready message=modelsAsAService is Managed and deployments are available OGXReady: status=True reason=AllDependentsHealthy message= RayReady: status=True reason= message= SparkOperatorReady: status=True reason=AllDependentsHealthy message= TrainerReady: status=True reason= message= TrustyAIReady: status=True reason= message= WorkbenchesReady: status=True reason=ReconcileSuccess message=Workbenches component is ready WorkbenchesV2Ready: status=False reason=Removed message=Submodule ManagementState is set to Removed --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 120m Ready 2026-09-07T09:51:34Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=False reason=Removed message=Monitoring is not enabled MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in Monitoring CR OpenTelemetryCollectorAvailable: status=False reason=MetricsAndTracesNotConfigured message=Metrics and traces are not configured in Monitoring CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in Monitoring CR NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR PersesAvailable: status=False reason=MetricsAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured ProvisioningSucceeded: status=True reason=Available message= Ready: status=True reason=Available message= --- Component CRs (Ready) --- NAME READY REASON VERSION aigateway.components.platform.opendatahub.io/default-aigateway True 1.26.2 NAME APPNAMESPACE AGE aihub.components.platform.opendatahub.io/default-aihub redhat-ods-applications 14m NAME PHASE READY REASON URL AGE dashboard.components.platform.opendatahub.io/default-dashboard Ready True AllDependentsHealthy https://rh-ai.apps.6bcf9958757301f6e7fb.prod.konflux-ocp-ci.dev/ 33m NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON mlflowoperator.components.platform.opendatahub.io/default-mlflowoperator True Ready NAME READY REASON DISTRIBUTION ogx.components.platform.opendatahub.io/default-ogx True AllDependentsHealthy Red Hat OpenShift AI NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON sparkoperator.components.platform.opendatahub.io/default-sparkoperator True AllDependentsHealthy NAME READY REASON AGE trainer.components.platform.opendatahub.io/default-trainer True 20m NAME READY REASON trustyai.components.platform.opendatahub.io/default-trustyai True NAME READY REASON PHASE workbenches.components.platform.opendatahub.io/default-workbenches True ReconcileSuccess Ready --- Deployments not fully ready (ODH/RHOAI namespaces) --- redhat-ods-applications/agent-ops-ui: ready=0/1 available=0 redhat-ods-applications/ai-gateway-operator: ready=0/1 available=0 redhat-ods-applications/aihub-controller-manager: ready=0/1 available=0 redhat-ods-applications/automl-ui: ready=0/1 available=0 redhat-ods-applications/autorag-ui: ready=0/1 available=0 redhat-ods-applications/catalog-controller-manager: ready=0/1 available=0 redhat-ods-applications/dashboard-operator: ready=0/2 available=0 redhat-ods-applications/dashboard-redirect: ready=0/2 available=0 redhat-ods-applications/data-science-pipelines-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/eval-hub-ui: ready=0/1 available=0 redhat-ods-applications/feast-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/gen-ai-ui: ready=0/1 available=0 redhat-ods-applications/kserve-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-localmodel-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-trainer-controller-manager: ready=0/1 available=0 redhat-ods-applications/kuberay-operator: ready=0/1 available=0 redhat-ods-applications/llmisvc-controller-manager: ready=0/1 available=0 redhat-ods-applications/maas-controller: ready=0/1 available=0 redhat-ods-applications/maas-ui: ready=0/1 available=0 redhat-ods-applications/mlflow-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/mlflow-ui: ready=0/1 available=0 redhat-ods-applications/model-registry-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-registry-ui: ready=0/1 available=0 redhat-ods-applications/model-serving-api: ready=0/1 available=0 redhat-ods-applications/notebook-controller-deployment: ready=0/1 available=0 redhat-ods-applications/notebooks-ui: ready=0/1 available=0 redhat-ods-applications/odh-model-controller: ready=0/1 available=0 redhat-ods-applications/odh-notebook-controller-manager: ready=0/1 available=0 redhat-ods-applications/ogx-k8s-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/opendatahub-feast-operator: ready=0/1 available=0 redhat-ods-applications/opendatahub-ogx-operator: ready=0/1 available=0 redhat-ods-applications/postgres-deployment: ready=0/1 available=0 redhat-ods-applications/rhods-dashboard: ready=0/2 available=0 redhat-ods-applications/spark-operator-controller: ready=0/1 available=0 redhat-ods-applications/spark-operator-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook: ready=0/1 available=0 redhat-ods-applications/trainer-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/trustyai-service-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/workbenches-operator: ready=0/1 available=0 redhat-ods-operator/rhods-operator: ready=0/3 available=0 rhoai-model-registries/model-catalog: ready=0/1 available=0 rhoai-model-registries/model-catalog-postgres: ready=0/1 available=0 --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- redhat-ods-applications/agent-ops-ui-7df989bc55-wlhzf: phase=Pending reason= redhat-ods-applications/ai-gateway-operator-89f69845b-w7969: phase=Pending reason= redhat-ods-applications/aihub-controller-manager-d877888b8-6sgnz: phase=Pending reason= redhat-ods-applications/automl-ui-74d9b97d49-fr8wz: phase=Pending reason= redhat-ods-applications/autorag-ui-58f548cbd-cm2s7: phase=Pending reason= redhat-ods-applications/catalog-controller-manager-67d887945b-rkzjh: phase=Pending reason= redhat-ods-applications/dashboard-operator-669d6564bd-c85d9: phase=Pending reason= redhat-ods-applications/dashboard-redirect-55f4444684-xgtjk: phase=Pending reason= redhat-ods-applications/eval-hub-ui-869b89d9b4-wvcsk: phase=Pending reason= redhat-ods-applications/feast-operator-controller-manager-5fc6bb8d67-7rlmx: phase=Pending reason= redhat-ods-applications/kserve-localmodel-controller-manager-fc8c879d7-tsw29: phase=Pending reason= redhat-ods-applications/kubeflow-trainer-controller-manager-5449979c85-zr9m6: phase=Pending reason= redhat-ods-applications/maas-controller-56c898799-rbr4l: phase=Pending reason= redhat-ods-applications/maas-ui-675f94cfd8-l7g85: phase=Pending reason= redhat-ods-applications/model-registry-operator-controller-manager-6bcdfb5b87-hqrtn: phase=Pending reason= redhat-ods-applications/notebook-controller-deployment-75c8689979-94b2h: phase=Pending reason= redhat-ods-applications/notebooks-ui-6577cc467-zfvzv: phase=Pending reason= redhat-ods-applications/odh-model-controller-6b9dd8b59d-bmf58: phase=Pending reason= redhat-ods-applications/ogx-k8s-operator-controller-manager-59777cdbbb-4lz7l: phase=Pending reason= redhat-ods-applications/rhods-dashboard-bfcc58b46-cmzzz: phase=Pending reason= redhat-ods-applications/rhods-dashboard-bfcc58b46-x29br: phase=Pending reason= redhat-ods-applications/spark-operator-controller-c594b967f-ds5f6: phase=Pending reason= redhat-ods-applications/spark-operator-module-controller-manager-59549b5854-hjqrv: phase=Pending reason= redhat-ods-applications/spark-operator-webhook-5d466b7d66-vnwd8: phase=Pending reason= redhat-ods-applications/workbenches-operator-c674d9c9-62cwk: phase=Pending reason= redhat-ods-operator/rhods-operator-b7665cfc6-2dpzd: phase=Pending reason= redhat-ods-operator/rhods-operator-b7665cfc6-6x65d: phase=Pending reason= rhoai-model-registries/model-catalog-64f9b98597-tv7r5: phase=Pending reason= rhoai-model-registries/model-catalog-postgres-768895d6c5-j4q6g: phase=Pending reason= --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/opendatahub --- (no events) --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m8s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m37s Warning FailedScheduling pod/agent-ops-ui-7df989bc55-wlhzf 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/agent-ops-ui-7df989bc55-wlhzf no nodes available to schedule pods 80s Warning FailedScheduling pod/agent-ops-ui-7df989bc55-wlhzf no nodes available to schedule pods 63m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 101m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 80s Warning FailedScheduling pod/model-registry-operator-controller-manager-6bcdfb5b87-hqrtn no nodes available to schedule pods 85s Warning FailedScheduling pod/model-registry-operator-controller-manager-6bcdfb5b87-hqrtn no nodes available to schedule pods 6m37s Warning FailedScheduling pod/model-registry-operator-controller-manager-6bcdfb5b87-hqrtn 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m59s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-w7969 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-w7969 no nodes available to schedule pods 110s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-w7969 no nodes available to schedule pods 110s Warning FailedScheduling pod/workbenches-operator-c674d9c9-62cwk no nodes available to schedule pods 115s Warning FailedScheduling pod/workbenches-operator-c674d9c9-62cwk no nodes available to schedule pods 6m59s Warning FailedScheduling pod/workbenches-operator-c674d9c9-62cwk 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m37s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-94b2h 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-94b2h no nodes available to schedule pods 80s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-94b2h no nodes available to schedule pods 6m37s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-6sgnz 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-6sgnz no nodes available to schedule pods 80s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-6sgnz no nodes available to schedule pods 6m38s Warning FailedScheduling pod/notebooks-ui-6577cc467-zfvzv 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/notebooks-ui-6577cc467-zfvzv no nodes available to schedule pods 80s Warning FailedScheduling pod/notebooks-ui-6577cc467-zfvzv no nodes available to schedule pods 105m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 101m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m8s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m58s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-bmf58 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-bmf58 no nodes available to schedule pods 110s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-bmf58 no nodes available to schedule pods 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m59s Warning FailedScheduling pod/automl-ui-74d9b97d49-fr8wz 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/automl-ui-74d9b97d49-fr8wz no nodes available to schedule pods 110s Warning FailedScheduling pod/automl-ui-74d9b97d49-fr8wz no nodes available to schedule pods 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 80s Warning FailedScheduling pod/maas-ui-675f94cfd8-l7g85 no nodes available to schedule pods 85s Warning FailedScheduling pod/maas-ui-675f94cfd8-l7g85 no nodes available to schedule pods 6m38s Warning FailedScheduling pod/maas-ui-675f94cfd8-l7g85 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 105m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m8s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 110s Warning FailedScheduling pod/spark-operator-webhook-5d466b7d66-vnwd8 no nodes available to schedule pods 115s Warning FailedScheduling pod/spark-operator-webhook-5d466b7d66-vnwd8 no nodes available to schedule pods 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m Warning FailedScheduling pod/spark-operator-webhook-5d466b7d66-vnwd8 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 16m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m59s Warning FailedScheduling pod/autorag-ui-58f548cbd-cm2s7 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/autorag-ui-58f548cbd-cm2s7 no nodes available to schedule pods 110s Warning FailedScheduling pod/autorag-ui-58f548cbd-cm2s7 no nodes available to schedule pods 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m38s Warning FailedScheduling pod/catalog-controller-manager-67d887945b-rkzjh 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/catalog-controller-manager-67d887945b-rkzjh no nodes available to schedule pods 80s Warning FailedScheduling pod/catalog-controller-manager-67d887945b-rkzjh no nodes available to schedule pods 110s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-hjqrv no nodes available to schedule pods 115s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-hjqrv no nodes available to schedule pods 6m59s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-hjqrv 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 52m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 101m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 25m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m8s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m37s Warning FailedScheduling pod/dashboard-operator-669d6564bd-c85d9 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/dashboard-operator-669d6564bd-c85d9 no nodes available to schedule pods 80s Warning FailedScheduling pod/dashboard-operator-669d6564bd-c85d9 no nodes available to schedule pods 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 101m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 105m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 104m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 110s Warning FailedScheduling pod/maas-controller-56c898799-rbr4l no nodes available to schedule pods 115s Warning FailedScheduling pod/maas-controller-56c898799-rbr4l no nodes available to schedule pods 6m58s Warning FailedScheduling pod/maas-controller-56c898799-rbr4l 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 101m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m37s Warning FailedScheduling pod/dashboard-redirect-55f4444684-xgtjk 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/dashboard-redirect-55f4444684-xgtjk no nodes available to schedule pods 80s Warning FailedScheduling pod/dashboard-redirect-55f4444684-xgtjk no nodes available to schedule pods 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 85m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 110s Warning FailedScheduling pod/spark-operator-controller-c594b967f-ds5f6 no nodes available to schedule pods 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 115s Warning FailedScheduling pod/spark-operator-controller-c594b967f-ds5f6 no nodes available to schedule pods 6m59s Warning FailedScheduling pod/spark-operator-controller-c594b967f-ds5f6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 80s Warning FailedScheduling pod/rhods-dashboard-bfcc58b46-x29br no nodes available to schedule pods 85s Warning FailedScheduling pod/rhods-dashboard-bfcc58b46-x29br no nodes available to schedule pods 6m37s Warning FailedScheduling pod/rhods-dashboard-bfcc58b46-x29br 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 52m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 34m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 33m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 19m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 18m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 110s Warning FailedScheduling pod/rhods-dashboard-bfcc58b46-cmzzz no nodes available to schedule pods 115s Warning FailedScheduling pod/rhods-dashboard-bfcc58b46-cmzzz no nodes available to schedule pods 6m59s Warning FailedScheduling pod/rhods-dashboard-bfcc58b46-cmzzz 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m58s Warning FailedScheduling pod/eval-hub-ui-869b89d9b4-wvcsk 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/eval-hub-ui-869b89d9b4-wvcsk no nodes available to schedule pods 110s Warning FailedScheduling pod/eval-hub-ui-869b89d9b4-wvcsk no nodes available to schedule pods 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m8s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m58s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-7rlmx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-7rlmx no nodes available to schedule pods 110s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-7rlmx no nodes available to schedule pods 80s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-zr9m6 no nodes available to schedule pods 85s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-zr9m6 no nodes available to schedule pods 6m38s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-zr9m6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m59s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-4lz7l 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-4lz7l no nodes available to schedule pods 110s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-4lz7l no nodes available to schedule pods 17m Warning FailedScheduling pod/kserve-localmodelnode-agent-zfds9 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 17m Warning FailedScheduling pod/kserve-localmodelnode-agent-vf2zp 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 19m Warning FailedScheduling pod/kserve-localmodelnode-agent-t6lz7 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 24m Warning FailedScheduling pod/kserve-localmodelnode-agent-n2qlk 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 24m Warning FailedScheduling pod/kserve-localmodelnode-agent-lwzt2 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 19m Warning FailedScheduling pod/kserve-localmodelnode-agent-8z62g 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 19m Warning FailedScheduling pod/kserve-localmodelnode-agent-8z62g 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 80s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-tsw29 no nodes available to schedule pods 85s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-tsw29 no nodes available to schedule pods 6m36s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-tsw29 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 119m Warning FailedMount pod/odh-observability-59db79cb46-jxh7c MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 119m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 119m Warning FailedMount pod/kuberay-operator-7d87f54798-z9q95 MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 119m Warning FailedMount pod/mlflow-operator-controller-manager-5bbb468f5f-4cnxp MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 119m Warning FailedMount pod/catalog-controller-manager-67d887945b-qlktz MountVolume.SetUp failed for volume "cert" : secret "catalog-webhook-service" not found 119m Warning FailedMount pod/model-registry-ui-67b8c946b9-s68np MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 119m Warning FailedMount pod/automl-ui-74d9b97d49-x268c MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 119m Warning FailedMount pod/mlflow-ui-7bb6d46f5f-nkqj6 MountVolume.SetUp failed for volume "proxy-tls" : secret "mlflow-proxy-tls" not found 118m Warning FailedMount pod/odh-notebook-controller-manager-5bcb7b86db-24424 MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 118m Warning FailedMount pod/trustyai-service-operator-controller-manager-d6bf47757-8djfr MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 117m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-b5ft6 Readiness probe failed: Get "https://10.134.0.38:8943/healthcheck": dial tcp 10.134.0.38:8943: connect: connection refused 117m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-b5ft6 Readiness probe failed: Get "https://10.134.0.38:8444/healthz": dial tcp 10.134.0.38:8444: connect: connection refused 117m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-b5ft6 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 117m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-2ncqt Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 117m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-2ncqt Readiness probe failed: Get "https://10.133.0.47:8943/healthcheck": dial tcp 10.133.0.47:8943: connect: connection refused 117m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-2ncqt Readiness probe failed: Get "https://10.133.0.47:8444/healthz": dial tcp 10.133.0.47:8444: connect: connection refused 101m Warning FailedMount pod/ogx-k8s-operator-controller-manager-59777cdbbb-x9blj MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 84m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 84m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 34m Warning FailedToRetrieveImagePullSecret pod/maas-controller-56c898799-t7zrv Unable to retrieve some image pull secrets (maas-controller-dockercfg-vp7c6); attempting to pull the image may not succeed. 34m Warning FailedMount pod/maas-controller-56c898799-t7zrv MountVolume.SetUp failed for volume "metrics-certs" : secret "maas-controller-metrics-tls" not found 34m Warning FailedMount pod/maas-controller-56c898799-t7zrv MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found 34m Warning Unhealthy pod/rhods-dashboard-bfcc58b46-v2xm8 Readiness probe failed: Get "https://10.132.0.55:8943/healthcheck": dial tcp 10.132.0.55:8943: connect: connection refused 34m Warning Unhealthy pod/kubeflow-trainer-controller-manager-5449979c85-h4tsl Readiness probe failed: Get "http://10.133.0.45:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 34m Warning Unhealthy pod/rhods-dashboard-bfcc58b46-v2xm8 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 34m Warning FailedToUpdateEndpoint endpoints/aihub-controller-manager-metrics-service Failed to update endpoint redhat-ods-applications/aihub-controller-manager-metrics-service: Operation cannot be fulfilled on endpoints "aihub-controller-manager-metrics-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/aihub-controller-manager-metrics-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: a5481786-dca0-4fe8-b208-e35728ace0fb, UID in object meta: 34m Warning Unhealthy pod/rhods-dashboard-bfcc58b46-42vcs Readiness probe failed: Get "https://10.133.0.52:8943/healthcheck": dial tcp 10.133.0.52:8943: connect: connection refused 34m Warning Unhealthy pod/rhods-dashboard-bfcc58b46-42vcs Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 33m Warning Unhealthy pod/rhods-dashboard-bfcc58b46-42vcs Readiness probe failed: Get "https://10.133.0.52:8444/healthz": dial tcp 10.133.0.52:8444: connect: connection refused 33m Warning Unhealthy pod/rhods-dashboard-bfcc58b46-v2xm8 Readiness probe failed: Get "https://10.132.0.55:8444/healthz": dial tcp 10.132.0.55:8444: connect: connection refused 33m Warning FailedMount pod/kuberay-operator-7d87f54798-z4s96 MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 33m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 33m Warning FailedMount pod/odh-model-controller-57457c4c89-8j6h9 MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 33m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 33m Warning FailedMount pod/kserve-controller-manager-5fbbd6dd88-wrr9b MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 32m Warning FailedMount pod/automl-ui-74d9b97d49-lzkxj MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 32m Warning FailedMount pod/autorag-ui-58f548cbd-8pn4g MountVolume.SetUp failed for volume "proxy-tls" : secret "autorag-proxy-tls" not found 31m Warning Unhealthy pod/spark-operator-controller-c594b967f-wpdtn Readiness probe failed: Get "http://10.133.0.108:8081/readyz": dial tcp 10.133.0.108:8081: connect: connection refused 31m Warning Unhealthy pod/spark-operator-webhook-5d466b7d66-g8r7p Readiness probe failed: Get "http://10.133.0.109:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 31m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-zhmtt Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 31m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-zhmtt Readiness probe failed: Get "https://10.134.0.106:8943/healthcheck": dial tcp 10.134.0.106:8943: connect: connection refused 31m Warning FailedToUpdateEndpoint endpoints/odh-dashboard-autorag-ui Failed to update endpoint redhat-ods-applications/odh-dashboard-autorag-ui: Operation cannot be fulfilled on endpoints "odh-dashboard-autorag-ui": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/odh-dashboard-autorag-ui, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 79bb98aa-3dac-4c9d-ae1e-46c4f3573e85, UID in object meta: 31m Warning FailedToUpdateEndpoint endpoints/odh-dashboard-automl-ui Failed to update endpoint redhat-ods-applications/odh-dashboard-automl-ui: Operation cannot be fulfilled on endpoints "odh-dashboard-automl-ui": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/odh-dashboard-automl-ui, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 9931d5c4-97af-4cd3-9b95-ee5288a56884, UID in object meta: 31m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-zhmtt Readiness probe failed: Get "https://10.134.0.106:8444/healthz": dial tcp 10.134.0.106:8444: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-7f94fcf99d-rwzbj Readiness probe failed: Get "https://10.133.0.114:8444/healthz": dial tcp 10.133.0.114:8444: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-7f94fcf99d-rwzbj Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 31m Warning Unhealthy pod/rhods-dashboard-7f94fcf99d-rwzbj Readiness probe failed: Get "https://10.133.0.114:8943/healthcheck": dial tcp 10.133.0.114:8943: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-dgpl5 Readiness probe failed: Get "https://10.133.0.111:8943/healthcheck": dial tcp 10.133.0.111:8943: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-dgpl5 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 30m Warning Unhealthy pod/rhods-dashboard-5c9d44b7b4-dgpl5 Readiness probe failed: Get "https://10.133.0.111:8444/healthz": dial tcp 10.133.0.111:8444: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-7f94fcf99d-6cjmj Readiness probe failed: Get "https://10.132.0.101:8943/healthcheck": dial tcp 10.132.0.101:8943: connect: connection refused 30m Warning Unhealthy pod/rhods-dashboard-7f94fcf99d-6cjmj Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 29m Warning Unhealthy pod/rhods-dashboard-7f94fcf99d-6cjmj Readiness probe failed: Get "https://10.132.0.101:8444/healthz": dial tcp 10.132.0.101:8444: connect: connection refused 27m Warning Unhealthy pod/odh-model-controller-57457c4c89-qkzqq Readiness probe failed: Get "http://10.132.0.107:8081/readyz": dial tcp 10.132.0.107:8081: connect: connection refused 26m Warning FailedMount pod/model-registry-operator-controller-manager-69d5948bc6-z5tm6 MountVolume.SetUp failed for volume "server-cert" : secret "model-registry-operator-controller-manager-metrics-service" not found 26m Warning FailedMount pod/model-registry-operator-controller-manager-69d5948bc6-z5tm6 MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 25m Warning FailedToUpdateEndpoint endpoints/aihub-controller-manager-metrics-service Failed to update endpoint redhat-ods-applications/aihub-controller-manager-metrics-service: Operation cannot be fulfilled on endpoints "aihub-controller-manager-metrics-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/aihub-controller-manager-metrics-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 10ee76ad-55dc-416d-b791-e304b7c8bcea, UID in object meta: 25m Warning Unhealthy pod/rhods-dashboard-775867476c-f7trw Readiness probe failed: Get "https://10.133.0.116:8943/healthcheck": dial tcp 10.133.0.116:8943: connect: connection refused 25m Warning Unhealthy pod/rhods-dashboard-775867476c-f7trw Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 25m Warning Unhealthy pod/rhods-dashboard-775867476c-f7trw Readiness probe failed: Get "https://10.133.0.116:8444/healthz": dial tcp 10.133.0.116:8444: connect: connection refused 25m Warning Unhealthy pod/rhods-dashboard-6c84498cbc-4cvv8 Readiness probe failed: Get "https://10.134.0.115:8943/healthcheck": dial tcp 10.134.0.115:8943: connect: connection refused 25m Warning Unhealthy pod/rhods-dashboard-6c84498cbc-4cvv8 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 25m Warning Unhealthy pod/rhods-dashboard-6c84498cbc-4cvv8 Readiness probe failed: Get "https://10.134.0.115:8444/healthz": dial tcp 10.134.0.115:8444: connect: connection refused 25m Warning Unhealthy pod/rhods-dashboard-6c84498cbc-vkd5b Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 25m Warning Unhealthy pod/rhods-dashboard-6c84498cbc-vkd5b Readiness probe failed: Get "https://10.132.0.110:8943/healthcheck": dial tcp 10.132.0.110:8943: connect: connection refused 24m Warning Unhealthy pod/rhods-dashboard-6c84498cbc-vkd5b Readiness probe failed: Get "https://10.132.0.110:8444/healthz": dial tcp 10.132.0.110:8444: connect: connection refused 24m Warning FailedMount pod/llmisvc-controller-manager-6df7b6c5d4-6vpxx MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 24m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 24m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 24m Warning FailedMount pod/odh-model-controller-57457c4c89-kt7sg MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 24m Warning Unhealthy pod/rhods-dashboard-775867476c-bpspj Readiness probe failed: Get "https://10.134.0.111:8444/healthz": dial tcp 10.134.0.111:8444: connect: connection refused 24m Warning Unhealthy pod/rhods-dashboard-775867476c-bpspj Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 24m Warning Unhealthy pod/rhods-dashboard-775867476c-bpspj Readiness probe failed: Get "https://10.134.0.111:8943/healthcheck": dial tcp 10.134.0.111:8943: connect: connection refused 23m Warning FailedToUpdateEndpoint endpoints/odh-dashboard-mlflow-ui Failed to update endpoint redhat-ods-applications/odh-dashboard-mlflow-ui: Operation cannot be fulfilled on endpoints "odh-dashboard-mlflow-ui": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/odh-dashboard-mlflow-ui, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: f0bffaf1-1c82-4e96-9561-689fb19b71e1, UID in object meta: 23m Warning FailedMount pod/trustyai-service-operator-controller-manager-d6bf47757-nmfrj MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 23m Warning Unhealthy pod/rhods-dashboard-85f74c5bcd-57z82 Readiness probe failed: Get "https://10.132.0.117:8444/healthz": dial tcp 10.132.0.117:8444: connect: connection refused 23m Warning Unhealthy pod/rhods-dashboard-85f74c5bcd-57z82 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 23m Warning Unhealthy pod/rhods-dashboard-85f74c5bcd-57z82 Readiness probe failed: Get "https://10.132.0.117:8943/healthcheck": dial tcp 10.132.0.117:8943: connect: connection refused 23m Warning Unhealthy pod/rhods-dashboard-775867476c-gp27m Readiness probe failed: Get "https://10.134.0.122:8444/healthz": dial tcp 10.134.0.122:8444: connect: connection refused 23m Warning Unhealthy pod/rhods-dashboard-775867476c-gp27m Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 23m Warning Unhealthy pod/rhods-dashboard-775867476c-gp27m Readiness probe failed: Get "https://10.134.0.122:8943/healthcheck": dial tcp 10.134.0.122:8943: connect: connection refused 22m Warning Unhealthy pod/rhods-dashboard-775867476c-2svc5 Readiness probe failed: Get "https://10.132.0.115:8943/healthcheck": dial tcp 10.132.0.115:8943: connect: connection refused 22m Warning Unhealthy pod/rhods-dashboard-775867476c-2svc5 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 22m Warning Unhealthy pod/rhods-dashboard-775867476c-2svc5 Readiness probe failed: Get "https://10.132.0.115:8444/healthz": dial tcp 10.132.0.115:8444: connect: connection refused 22m Warning Unhealthy pod/rhods-dashboard-85f74c5bcd-vhxz2 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 22m Warning Unhealthy pod/rhods-dashboard-85f74c5bcd-vhxz2 Readiness probe failed: Get "https://10.134.0.119:8943/healthcheck": dial tcp 10.134.0.119:8943: connect: connection refused 21m Warning Unhealthy pod/rhods-dashboard-85f74c5bcd-vhxz2 Readiness probe failed: Get "https://10.134.0.119:8444/healthz": dial tcp 10.134.0.119:8444: connect: connection refused 19m Warning FailedMount pod/odh-model-controller-57457c4c89-bjb7h MountVolume.SetUp failed for volume "metrics-certs" : secret "odh-model-controller-metrics-tls" not found 19m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 19m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 19m Warning Unhealthy pod/rhods-dashboard-7495c45f5c-dt7qv Readiness probe failed: Get "https://10.134.0.125:8943/healthcheck": dial tcp 10.134.0.125:8943: connect: connection refused 19m Warning Unhealthy pod/rhods-dashboard-7495c45f5c-dt7qv Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 19m Warning Unhealthy pod/rhods-dashboard-7495c45f5c-dt7qv Readiness probe failed: Get "https://10.134.0.125:8444/healthz": dial tcp 10.134.0.125:8444: connect: connection refused 19m Warning Unhealthy pod/rhods-dashboard-7495c45f5c-ssd86 Readiness probe failed: Get "https://10.132.0.123:8943/healthcheck": dial tcp 10.132.0.123:8943: connect: connection refused 19m Warning Unhealthy pod/rhods-dashboard-7495c45f5c-ssd86 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 18m Warning Unhealthy pod/model-serving-api-7df8c9655f-n89d9 Readiness probe failed: Get "https://10.134.0.133:8443/readyz": dial tcp 10.134.0.133:8443: connect: connection refused 18m Warning Unhealthy pod/kserve-module-controller-manager-66ff7c977-4fxv8 Readiness probe failed: Get "http://10.134.0.129:8081/readyz": dial tcp 10.134.0.129:8081: connect: connection refused 18m Warning Unhealthy pod/rhods-dashboard-7495c45f5c-ssd86 Readiness probe failed: Get "https://10.132.0.123:8444/healthz": dial tcp 10.132.0.123:8444: connect: connection refused 17m Warning FailedMount pod/kserve-controller-manager-5fbbd6dd88-xt6c2 MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 17m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 17m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 16m Warning FailedMount pod/mlflow-operator-controller-manager-5bbb468f5f-phrs7 MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 16m Warning FailedMount pod/mlflow-ui-7bb6d46f5f-sqwq5 MountVolume.SetUp failed for volume "proxy-tls" : secret "mlflow-proxy-tls" not found 15m Warning Unhealthy pod/rhods-dashboard-775867476c-6thcb Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 15m Warning Unhealthy pod/rhods-dashboard-775867476c-6thcb Readiness probe failed: Get "https://10.134.0.128:8943/healthcheck": dial tcp 10.134.0.128:8943: connect: connection refused 15m Warning Unhealthy pod/rhods-dashboard-775867476c-6thcb Readiness probe failed: Get "https://10.134.0.128:8444/healthz": dial tcp 10.134.0.128:8444: connect: connection refused 14m Warning FailedMount pod/kuberay-operator-7d87f54798-rbxgx MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 14m Warning FailedMount pod/autorag-ui-58f548cbd-ftf9r MountVolume.SetUp failed for volume "proxy-tls" : secret "autorag-proxy-tls" not found 14m Warning FailedMount pod/automl-ui-74d9b97d49-md9l4 MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 14m Warning Unhealthy pod/rhods-dashboard-775867476c-pg5h8 Readiness probe failed: Get "https://10.132.0.126:8943/healthcheck": dial tcp 10.132.0.126:8943: connect: connection refused 14m Warning Unhealthy pod/rhods-dashboard-775867476c-pg5h8 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 14m Warning Unhealthy pod/rhods-dashboard-775867476c-pg5h8 Readiness probe failed: Get "https://10.132.0.126:8444/healthz": dial tcp 10.132.0.126:8444: connect: connection refused 14m Warning FailedMount pod/model-registry-operator-controller-manager-6bcdfb5b87-5knpg MountVolume.SetUp failed for volume "server-cert" : secret "model-registry-operator-controller-manager-metrics-service" not found 14m Warning FailedMount pod/maas-controller-56c898799-jhs7l MountVolume.SetUp failed for volume "metrics-certs" : secret "maas-controller-metrics-tls" not found 14m Warning FailedMount pod/maas-controller-56c898799-jhs7l MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found 14m Warning FailedMount pod/ogx-k8s-operator-controller-manager-59777cdbbb-lknrb MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 13m Warning Unhealthy pod/rhods-dashboard-85f74c5bcd-lj9bf Readiness probe failed: Get "https://10.132.0.132:8943/healthcheck": dial tcp 10.132.0.132:8943: connect: connection refused 13m Warning Unhealthy pod/rhods-dashboard-85f74c5bcd-lj9bf Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 13m Warning Unhealthy pod/rhods-dashboard-85f74c5bcd-lj9bf Readiness probe failed: Get "https://10.132.0.132:8444/healthz": dial tcp 10.132.0.132:8444: connect: connection refused 13m Warning Unhealthy pod/rhods-dashboard-85f74c5bcd-5n9z7 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 13m Warning Unhealthy pod/rhods-dashboard-85f74c5bcd-5n9z7 Readiness probe failed: Get "https://10.134.0.142:8943/healthcheck": dial tcp 10.134.0.142:8943: connect: connection refused 12m Warning Unhealthy pod/rhods-dashboard-85f74c5bcd-5n9z7 Readiness probe failed: Get "https://10.134.0.142:8444/healthz": dial tcp 10.134.0.142:8444: connect: connection refused 8m43s Warning Unhealthy pod/mlflow-758c97497-sgz2j Readiness probe failed: Get "https://10.132.0.139:8443/mlflow/api/3.0/mlflow/server-info": dial tcp 10.132.0.139:8443: connect: connection refused 8m42s Warning Unhealthy pod/kserve-controller-manager-5fbbd6dd88-xt6c2 Readiness probe failed: Get "http://10.134.0.136:8081/readyz": dial tcp 10.134.0.136:8081: connect: connection refused 8m42s Warning Unhealthy pod/trainer-operator-controller-manager-6b585cbb9f-jsjbq Readiness probe failed: Get "http://10.134.0.130:8081/readyz": dial tcp 10.134.0.130:8081: connect: connection refused 8m42s Warning Unhealthy pod/llmisvc-controller-manager-6df7b6c5d4-pwbvv Readiness probe failed: Get "http://10.134.0.137:8081/readyz": dial tcp 10.134.0.137:8081: connect: connection refused 7m54s Warning NodeNotReady pod/notebooks-ui-6577cc467-6fxnb Node is not ready 7m54s Warning NodeNotReady pod/kserve-localmodelnode-agent-zfds9 Node is not ready 7m54s Warning NodeNotReady pod/agent-ops-ui-7df989bc55-wh2xr Node is not ready 7m54s Warning NodeNotReady pod/dashboard-redirect-55f4444684-zpwrc Node is not ready 7m54s Warning NodeNotReady pod/data-science-pipelines-operator-controller-manager-56b8f89fzdtv Node is not ready 7m54s Warning NodeNotReady pod/dashboard-operator-669d6564bd-gcllm Node is not ready 7m54s Warning NodeNotReady pod/aihub-controller-manager-d877888b8-5w7hh Node is not ready 7m53s Warning NodeNotReady pod/mlflow-ui-7bb6d46f5f-sqwq5 Node is not ready 7m53s Warning NodeNotReady pod/dashboard-redirect-55f4444684-bs9jf Node is not ready 7m53s Warning NodeNotReady pod/kubeflow-trainer-controller-manager-5449979c85-cjd4x Node is not ready 7m53s Warning NodeNotReady pod/spark-operator-controller-c594b967f-ltf87 Node is not ready 7m53s Warning NodeNotReady pod/dashboard-operator-669d6564bd-twg9s Node is not ready 7m53s Warning NodeNotReady pod/maas-controller-56c898799-jhs7l Node is not ready 7m53s Warning NodeNotReady pod/spark-operator-webhook-5d466b7d66-rtwvj Node is not ready 7m53s Warning NodeNotReady pod/gen-ai-ui-6459ffd979-lqcdt Node is not ready 7m53s Warning NodeNotReady pod/rhods-dashboard-bfcc58b46-jng4q Node is not ready 7m53s Warning NodeNotReady pod/ai-gateway-operator-89f69845b-z89qb Node is not ready 7m53s Warning NodeNotReady pod/mlflow-operator-controller-manager-5bbb468f5f-phrs7 Node is not ready 7m53s Warning NodeNotReady pod/model-registry-ui-67b8c946b9-tcpf7 Node is not ready 7m53s Warning NodeNotReady pod/kuberay-operator-7d87f54798-rbxgx Node is not ready 7m53s Warning NodeNotReady pod/ogx-k8s-operator-controller-manager-59777cdbbb-lknrb Node is not ready 7m53s Warning NodeNotReady pod/kserve-localmodelnode-agent-vf2zp Node is not ready 7m53s Warning NodeNotReady pod/trustyai-service-operator-controller-manager-d6bf47757-vc2l5 Node is not ready 7m53s Warning NodeNotReady pod/opendatahub-feast-operator-84ff599545-c8vcj Node is not ready 7m53s Warning NodeNotReady pod/catalog-controller-manager-67d887945b-rnlnt Node is not ready 7m53s Warning NodeNotReady pod/feast-operator-controller-manager-5fc6bb8d67-zh5s9 Node is not ready 7m53s Warning NodeNotReady pod/spark-operator-module-controller-manager-59549b5854-ksk2n Node is not ready 7m53s Warning NodeNotReady pod/postgres-deployment-86754cb759-jpl2b Node is not ready 7m53s Warning NodeNotReady pod/model-registry-operator-controller-manager-6bcdfb5b87-5knpg Node is not ready 7m53s Warning NodeNotReady pod/automl-ui-74d9b97d49-md9l4 Node is not ready 7m52s Warning NodeNotReady pod/kserve-localmodel-controller-manager-fc8c879d7-djt7x Node is not ready 7m52s Warning NodeNotReady pod/notebook-controller-deployment-75c8689979-sfs7d Node is not ready 7m52s Warning NodeNotReady pod/maas-ui-675f94cfd8-57bsm Node is not ready 7m52s Warning NodeNotReady pod/model-serving-api-7df8c9655f-7rjb7 Node is not ready 7m52s Warning NodeNotReady pod/opendatahub-ogx-operator-644d45d5b-scmw7 Node is not ready 7m52s Warning NodeNotReady pod/llmisvc-controller-manager-6df7b6c5d4-pwbvv Node is not ready 7m52s Warning NodeNotReady pod/eval-hub-ui-869b89d9b4-vrvvl Node is not ready 7m52s Warning NodeNotReady pod/trainer-operator-controller-manager-6b585cbb9f-jsjbq Node is not ready 7m52s Warning NodeNotReady pod/rhods-dashboard-bfcc58b46-7d2kb Node is not ready 7m52s Warning NodeNotReady pod/odh-model-controller-6b9dd8b59d-rql4v Node is not ready 7m52s Warning NodeNotReady pod/kserve-controller-manager-5fbbd6dd88-xt6c2 Node is not ready 7m52s Warning NodeNotReady pod/workbenches-operator-c674d9c9-wps6f Node is not ready 7m52s Warning NodeNotReady pod/odh-notebook-controller-manager-5bcb7b86db-fwz24 Node is not ready 7m52s Warning NodeNotReady pod/kserve-module-controller-manager-66ff7c977-l8vxt Node is not ready 7m52s Warning NodeNotReady pod/autorag-ui-58f548cbd-ftf9r Node is not ready 6m50s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: skipping Pod rhods-dashboard-bfcc58b46-jng4q for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-128-124.ec2.internal Not Found 2m7s Warning FailedCreate replicaset/kserve-module-controller-manager-66ff7c977 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m5s Warning FailedCreate replicaset/llmisvc-controller-manager-6df7b6c5d4 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m5s Warning FailedCreate replicaset/model-serving-api-7df8c9655f Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m2s Warning FailedCreate replicaset/model-registry-ui-67b8c946b9 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m1s Warning FailedCreate replicaset/data-science-pipelines-operator-controller-manager-56b8f89dc6 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m Warning FailedCreate replicaset/kserve-controller-manager-5fbbd6dd88 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 110s Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-d6bf47757 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 109s Warning FailedCreate replicaset/gen-ai-ui-6459ffd979 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 109s Warning FailedCreate replicaset/opendatahub-ogx-operator-644d45d5b Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 108s Warning FailedCreate replicaset/mlflow-operator-controller-manager-5bbb468f5f Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 108s Warning FailedCreate replicaset/odh-notebook-controller-manager-5bcb7b86db Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 108s Warning FailedCreate replicaset/mlflow-ui-7bb6d46f5f Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 107s Warning FailedCreate replicaset/opendatahub-feast-operator-84ff599545 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 107s Warning FailedCreate replicaset/trainer-operator-controller-manager-6b585cbb9f Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 106s Warning FailedCreate replicaset/kuberay-operator-7d87f54798 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 105s Warning FailedCreate replicaset/postgres-deployment-86754cb759 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 85s Warning FailedCreate replicaset/dashboard-redirect-55f4444684 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 85s Warning FailedCreate replicaset/dashboard-operator-669d6564bd Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- LAST SEEN TYPE REASON OBJECT MESSAGE 116s Warning FailedScheduling pod/rhods-operator-b7665cfc6-6x65d no nodes available to schedule pods 7m Warning FailedScheduling pod/rhods-operator-b7665cfc6-6x65d 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 6m37s Warning FailedScheduling pod/rhods-operator-b7665cfc6-2dpzd 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85s Warning FailedScheduling pod/rhods-operator-b7665cfc6-2dpzd no nodes available to schedule pods 81s Warning FailedScheduling pod/rhods-operator-b7665cfc6-2dpzd no nodes available to schedule pods 111s Warning FailedScheduling pod/rhods-operator-b7665cfc6-6x65d no nodes available to schedule pods 120m Warning AppliedWithWarnings installplan/install-q4p94 1 warning(s) generated during installation of operator "rhods-operator.3.6.0-ea.1" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 7m54s Warning NodeNotReady pod/rhods-operator-b7665cfc6-nzzsj Node is not ready 7m54s Warning NodeNotReady pod/rhods-operator-b7665cfc6-42qgk Node is not ready 7m54s Warning NodeNotReady pod/rhods-operator-b7665cfc6-k8477 Node is not ready 7m52s Warning ComponentUnhealthy clusterserviceversion/rhods-operator.3.6.0-ea.1 installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. 6m51s Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: skipping Pod rhods-operator-b7665cfc6-nzzsj for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-128-124.ec2.internal Not Found 6m51s Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: skipping Pod rhods-operator-b7665cfc6-nzzsj for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-128-124.ec2.internal Not Found 2m50s Warning InstallCheckFailed clusterserviceversion/rhods-operator.3.6.0-ea.1 install timeout 86s Warning FailedCreate replicaset/rhods-operator-b7665cfc6 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found --- namespace/rhoai-model-registries --- LAST SEEN TYPE REASON OBJECT MESSAGE 87m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 87m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 88m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 111s Warning FailedScheduling pod/model-catalog-postgres-768895d6c5-j4q6g no nodes available to schedule pods 116s Warning FailedScheduling pod/model-catalog-postgres-768895d6c5-j4q6g no nodes available to schedule pods 7m Warning FailedScheduling pod/model-catalog-postgres-768895d6c5-j4q6g 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 88m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 116s Warning FailedScheduling pod/model-catalog-64f9b98597-tv7r5 no nodes available to schedule pods 6m59s Warning FailedScheduling pod/model-catalog-64f9b98597-tv7r5 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 111s Warning FailedScheduling pod/model-catalog-64f9b98597-tv7r5 no nodes available to schedule pods 118m Warning FailedMount pod/model-catalog-db9768684-nlbnk MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 118m Warning FailedMount pod/model-catalog-db9768684-nlbnk MountVolume.SetUp failed for volume "kube-rbac-proxy-tls" : secret "model-catalog-kube-rbac-proxy" not found 88m Warning Unhealthy pod/db-model-registry0-865b4f5d6f-kbrmt Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 33m Warning Unhealthy pod/model-catalog-postgres-58445c95fb-2frh6 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 33m Warning Unhealthy pod/model-catalog-db9768684-nlbnk Readiness probe failed: Get "http://10.132.0.58:8080/readyz": dial tcp 10.132.0.58:8080: connect: connection refused 33m Warning Unhealthy pod/model-catalog-db9768684-nlbnk Readiness probe failed: Get "https://10.132.0.58:8888/healthz": dial tcp 10.132.0.58:8888: connect: connection refused 14m Warning FailedMount pod/model-catalog-64f9b98597-4v99f MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 13m Warning Unhealthy pod/model-catalog-64f9b98597-4v99f Readiness probe failed: Get "http://10.133.0.138:8080/readyz": dial tcp 10.133.0.138:8080: connect: connection refused 7m54s Warning NodeNotReady pod/model-catalog-64f9b98597-4v99f Node is not ready 7m54s Warning NodeNotReady pod/model-catalog-postgres-768895d6c5-gdx4v Node is not ready --- namespace/rhods-notebooks --- (no events) --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-09-07T09:20:58Z) (no matching error/webhook lines in operator logs for this time window) --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-b7665cfc6-2dpzd.log ==================== ==================== OPERATOR POD LOG: rhods-operator-b7665cfc6-6x65d.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/agent-ops-ui-7df989bc55-wlhzf.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/ai-gateway-operator-89f69845b-w7969.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/aihub-controller-manager-d877888b8-6sgnz.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/automl-ui-74d9b97d49-fr8wz.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/autorag-ui-58f548cbd-cm2s7.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/catalog-controller-manager-67d887945b-rkzjh.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-operator-669d6564bd-c85d9.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-55f4444684-xgtjk.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/eval-hub-ui-869b89d9b4-wvcsk.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/feast-operator-controller-manager-5fc6bb8d67-7rlmx.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-localmodel-controller-manager-fc8c879d7-tsw29.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kubeflow-trainer-controller-manager-5449979c85-zr9m6.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-controller-56c898799-rbr4l.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-ui-675f94cfd8-l7g85.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-registry-operator-controller-manager-6bcdfb5b87-hqrtn.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/notebook-controller-deployment-75c8689979-94b2h.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/notebooks-ui-6577cc467-zfvzv.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-model-controller-6b9dd8b59d-bmf58.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/ogx-k8s-operator-controller-manager-59777cdbbb-4lz7l.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/rhods-dashboard-bfcc58b46-x29br.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-controller-c594b967f-ds5f6.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-module-controller-manager-59549b5854-hjqrv.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-webhook-5d466b7d66-vnwd8.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/workbenches-operator-c674d9c9-62cwk.log ==================== ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-64f9b98597-tv7r5.log ==================== ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-postgres-768895d6c5-j4q6g.log ====================