==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== ResolutionFailed: status=True reason=ErrorPreventedResolution message=error using catalogsource openshift-marketplace/rhoai-catalog-dev: failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF" redhat-ods-applications/agent-ops-ui: ready=0/1 available=0 redhat-ods-applications/ai-gateway-operator: ready=0/1 available=0 redhat-ods-applications/aihub-controller-manager: ready=0/1 available=0 redhat-ods-applications/automl-ui: ready=0/1 available=0 redhat-ods-applications/autorag-ui: ready=0/1 available=0 redhat-ods-applications/catalog-controller-manager: ready=0/1 available=0 redhat-ods-applications/dashboard-operator: ready=0/2 available=0 redhat-ods-applications/dashboard-redirect: ready=0/2 available=0 redhat-ods-applications/data-science-pipelines-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/eval-hub-ui: ready=0/1 available=0 redhat-ods-applications/feast-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/gen-ai-ui: ready=0/1 available=0 redhat-ods-applications/kserve-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-localmodel-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-trainer-controller-manager: ready=0/1 available=0 redhat-ods-applications/kuberay-operator: ready=0/1 available=0 redhat-ods-applications/llmisvc-controller-manager: ready=0/1 available=0 redhat-ods-applications/maas-controller: ready=0/1 available=0 redhat-ods-applications/maas-ui: ready=0/1 available=0 redhat-ods-applications/mlflow: ready=0/1 available=0 redhat-ods-applications/mlflow-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/mlflow-ui: ready=0/1 available=0 redhat-ods-applications/model-registry-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-registry-ui: ready=0/1 available=0 redhat-ods-applications/model-serving-api: ready=0/1 available=0 redhat-ods-applications/notebook-controller-deployment: ready=0/1 available=0 redhat-ods-applications/notebooks-ui: ready=0/1 available=0 redhat-ods-applications/odh-model-controller: ready=0/1 available=0 redhat-ods-applications/odh-notebook-controller-manager: ready=0/1 available=0 redhat-ods-applications/ogx-k8s-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/opendatahub-feast-operator: ready=0/1 available=0 redhat-ods-applications/opendatahub-ogx-operator: ready=0/1 available=0 redhat-ods-applications/postgres-deployment: ready=0/1 available=0 redhat-ods-applications/rhods-dashboard: ready=0/2 available=0 redhat-ods-applications/spark-operator-controller: ready=0/1 available=0 redhat-ods-applications/spark-operator-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook: ready=0/1 available=0 redhat-ods-applications/trainer-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/trustyai-service-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/workbenches-operator: ready=0/1 available=0 redhat-ods-operator/rhods-operator: ready=0/3 available=0 rhoai-model-registries/model-catalog: ready=0/1 available=0 rhoai-model-registries/model-catalog-postgres: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook-5d466b7d66-jpq9x: phase=Pending reason= 10m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 118s Warning FailedScheduling pod/trainer-operator-controller-manager-6b585cbb9f-dq7fg no nodes available to schedule pods 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-fknxk no nodes available to schedule pods 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 118s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-fknxk no nodes available to schedule pods 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m17s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-fknxk 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/mlflow-ui-7bb6d46f5f-wd6r5 no nodes available to schedule pods 7m17s Warning FailedScheduling pod/agent-ops-ui-7f9fbf68-msrj6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/agent-ops-ui-7f9fbf68-msrj6 no nodes available to schedule pods 113s Warning FailedScheduling pod/agent-ops-ui-7f9fbf68-msrj6 no nodes available to schedule pods 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m16s Warning FailedScheduling pod/mlflow-ui-7bb6d46f5f-wd6r5 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m17s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-gqxm5 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-gqxm5 no nodes available to schedule pods 113s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-gqxm5 no nodes available to schedule pods 82m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114s Warning FailedScheduling pod/notebooks-ui-6577cc467-n78h6 no nodes available to schedule pods 118s Warning FailedScheduling pod/notebooks-ui-6577cc467-n78h6 no nodes available to schedule pods 7m17s Warning FailedScheduling pod/notebooks-ui-6577cc467-n78h6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m24s Warning FailedScheduling pod/maas-controller-56c898799-x8jbb no nodes available to schedule pods 2m28s Warning FailedScheduling pod/maas-controller-56c898799-x8jbb no nodes available to schedule pods 7m41s Warning FailedScheduling pod/maas-controller-56c898799-x8jbb 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m17s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-wsx4s 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-wsx4s no nodes available to schedule pods 113s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-wsx4s no nodes available to schedule pods 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 10m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m57s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 21m Warning FailedScheduling pod/kserve-localmodelnode-agent-927tf 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 21m Warning FailedScheduling pod/kserve-localmodelnode-agent-927tf 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 83m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-6fn72 no nodes available to schedule pods 118s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-6fn72 no nodes available to schedule pods 7m18s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-6fn72 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 113s Warning FailedScheduling pod/llmisvc-controller-manager-6df7b6c5d4-qksnx no nodes available to schedule pods 118s Warning FailedScheduling pod/llmisvc-controller-manager-6df7b6c5d4-qksnx no nodes available to schedule pods 7m16s Warning FailedScheduling pod/llmisvc-controller-manager-6df7b6c5d4-qksnx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 29m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 113s Warning FailedScheduling pod/trainer-operator-controller-manager-6b585cbb9f-dq7fg no nodes available to schedule pods 7m16s Warning FailedScheduling pod/trainer-operator-controller-manager-6b585cbb9f-dq7fg 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 30m Warning FailedScheduling pod/kserve-localmodelnode-agent-mmmpl 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m39s Warning FailedScheduling pod/catalog-controller-manager-76985cd459-dnvc8 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2m28s Warning FailedScheduling pod/catalog-controller-manager-76985cd459-dnvc8 no nodes available to schedule pods 2m23s Warning FailedScheduling pod/catalog-controller-manager-76985cd459-dnvc8 no nodes available to schedule pods 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 22m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 22m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 21m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m24s Warning FailedScheduling pod/spark-operator-webhook-5d466b7d66-jpq9x no nodes available to schedule pods 2m28s Warning FailedScheduling pod/spark-operator-webhook-5d466b7d66-jpq9x no nodes available to schedule pods 7m40s Warning FailedScheduling pod/spark-operator-webhook-5d466b7d66-jpq9x 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 17m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m39s Warning FailedScheduling pod/postgres-deployment-86754cb759-4zt6s 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 30m Warning FailedScheduling pod/kserve-localmodelnode-agent-nh54p 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 22m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 21m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 10m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m57s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 21m Warning FailedScheduling pod/kserve-localmodelnode-agent-nmql4 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m18s Warning FailedScheduling pod/dashboard-operator-669d6564bd-r5nqx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/dashboard-operator-669d6564bd-r5nqx no nodes available to schedule pods 114s Warning FailedScheduling pod/dashboard-operator-669d6564bd-r5nqx no nodes available to schedule pods 7m39s Warning FailedScheduling pod/dashboard-operator-669d6564bd-z5xvv 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2m28s Warning FailedScheduling pod/dashboard-operator-669d6564bd-z5xvv no nodes available to schedule pods 2m23s Warning FailedScheduling pod/dashboard-operator-669d6564bd-z5xvv no nodes available to schedule pods 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m17s Warning FailedScheduling pod/rhods-dashboard-78d9976d4f-f8w8s 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7m40s Warning FailedScheduling pod/model-registry-operator-controller-manager-758fcb8c6b-qjrvp 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m28s Warning FailedScheduling pod/model-registry-operator-controller-manager-758fcb8c6b-qjrvp no nodes available to schedule pods 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m24s Warning FailedScheduling pod/model-registry-operator-controller-manager-758fcb8c6b-qjrvp no nodes available to schedule pods 21m Warning FailedScheduling pod/kserve-localmodelnode-agent-nmql4 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 13m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 118s Warning FailedScheduling pod/rhods-dashboard-78d9976d4f-f8w8s no nodes available to schedule pods 114s Warning FailedScheduling pod/rhods-dashboard-78d9976d4f-f8w8s no nodes available to schedule pods 57m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m39s Warning FailedScheduling pod/dashboard-redirect-76d74cfc9f-fznfl 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2m28s Warning FailedScheduling pod/dashboard-redirect-76d74cfc9f-fznfl no nodes available to schedule pods 2m23s Warning FailedScheduling pod/dashboard-redirect-76d74cfc9f-fznfl no nodes available to schedule pods 7m40s Warning FailedScheduling pod/rhods-dashboard-78d9976d4f-jd7mm 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 11m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning FailedScheduling pod/kserve-localmodelnode-agent-rw779 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 47m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m28s Warning FailedScheduling pod/rhods-dashboard-78d9976d4f-jd7mm no nodes available to schedule pods 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m24s Warning FailedScheduling pod/rhods-dashboard-78d9976d4f-jd7mm no nodes available to schedule pods 30m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 29m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning FailedScheduling pod/kserve-localmodelnode-agent-s8gsw 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 21m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 10m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m17s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f89khdkb 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f89khdkb no nodes available to schedule pods 114s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f89khdkb no nodes available to schedule pods 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114s Warning FailedScheduling pod/model-serving-api-7df8c9655f-2qdks no nodes available to schedule pods 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m57s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 118s Warning FailedScheduling pod/model-serving-api-7df8c9655f-2qdks no nodes available to schedule pods 7m17s Warning FailedScheduling pod/model-serving-api-7df8c9655f-2qdks 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 10m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m57s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m18s Warning FailedScheduling pod/maas-ui-74675d48f9-cxmkg 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/maas-ui-74675d48f9-cxmkg no nodes available to schedule pods 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m39s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-ncl4z 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114s Warning FailedScheduling pod/maas-ui-74675d48f9-cxmkg no nodes available to schedule pods 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-cnhbk no nodes available to schedule pods 118s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-cnhbk no nodes available to schedule pods 7m18s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-cnhbk 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 84m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m40s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-nb66k 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2m28s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-nb66k no nodes available to schedule pods 2m24s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-nb66k no nodes available to schedule pods 2m24s Warning FailedScheduling pod/spark-operator-controller-c594b967f-ffppx no nodes available to schedule pods 2m28s Warning FailedScheduling pod/spark-operator-controller-c594b967f-ffppx no nodes available to schedule pods 7m40s Warning FailedScheduling pod/spark-operator-controller-c594b967f-ffppx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 31m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m57s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 113s Warning FailedScheduling pod/mlflow-ui-7bb6d46f5f-wd6r5 no nodes available to schedule pods 82m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m39s Warning FailedScheduling pod/mlflow-6f9ddc956d-6m4z9 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 49m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m18s Warning FailedScheduling pod/gen-ai-ui-7c7dbc9d78-9dh9f 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/gen-ai-ui-7c7dbc9d78-9dh9f no nodes available to schedule pods 114s Warning FailedScheduling pod/gen-ai-ui-7c7dbc9d78-9dh9f no nodes available to schedule pods 113s Warning FailedScheduling pod/kserve-module-controller-manager-66ff7c977-nhf2c no nodes available to schedule pods 118s Warning FailedScheduling pod/kserve-module-controller-manager-66ff7c977-nhf2c no nodes available to schedule pods 7m17s Warning FailedScheduling pod/kserve-module-controller-manager-66ff7c977-nhf2c 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 57m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m23s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-ncl4z no nodes available to schedule pods 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m28s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-ncl4z no nodes available to schedule pods 7m18s Warning FailedScheduling pod/mlflow-operator-controller-manager-5bbb468f5f-btsd8 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/mlflow-operator-controller-manager-5bbb468f5f-btsd8 no nodes available to schedule pods 114s Warning FailedScheduling pod/mlflow-operator-controller-manager-5bbb468f5f-btsd8 no nodes available to schedule pods 7m17s Warning FailedScheduling pod/kserve-controller-manager-5fbbd6dd88-b729h 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/kserve-controller-manager-5fbbd6dd88-b729h no nodes available to schedule pods 113s Warning FailedScheduling pod/kserve-controller-manager-5fbbd6dd88-b729h no nodes available to schedule pods 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 117m Warning FailedMount pod/odh-observability-59db79cb46-8nd4t MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 117m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 117m Warning FailedMount pod/kuberay-operator-7d87f54798-dkgpd MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 117m Warning FailedMount pod/model-registry-operator-controller-manager-758fcb8c6b-c9wvs MountVolume.SetUp failed for volume "server-cert" : secret "model-registry-operator-controller-manager-metrics-service" not found 117m Warning FailedMount pod/model-registry-operator-controller-manager-758fcb8c6b-c9wvs MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 117m Warning FailedMount pod/mlflow-operator-controller-manager-5bbb468f5f-8ww9r MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 117m Warning FailedMount pod/notebooks-ui-6577cc467-55hxj MountVolume.SetUp failed for volume "proxy-tls" : secret "notebooks-proxy-tls" not found 117m Warning FailedMount pod/eval-hub-ui-869b89d9b4-x7vxb MountVolume.SetUp failed for volume "proxy-tls" : secret "eval-hub-proxy-tls" not found 117m Warning FailedMount pod/model-registry-ui-7897899d9b-ls82g MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 117m Warning FailedMount pod/odh-notebook-controller-manager-76dd46fbc-zl85r MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 117m Warning FailedMount pod/mlflow-ui-7bb6d46f5f-l56tj MountVolume.SetUp failed for volume "proxy-tls" : secret "mlflow-proxy-tls" not found 115m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-xkqfs Readiness probe failed: Get "https://10.133.0.55:8943/healthcheck": dial tcp 10.133.0.55:8943: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-xkqfs Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 115m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-xkqfs Readiness probe failed: Get "https://10.133.0.55:8444/healthz": dial tcp 10.133.0.55:8444: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-x2vhh Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 115m Warning FailedMount pod/maas-controller-56c898799-jxzbq MountVolume.SetUp failed for volume "metrics-certs" : secret "maas-controller-metrics-tls" not found 115m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-x2vhh Readiness probe failed: Get "https://10.132.0.47:8943/healthcheck": dial tcp 10.132.0.47:8943: connect: connection refused 115m Warning FailedMount pod/maas-controller-56c898799-jxzbq MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found 115m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-x2vhh Readiness probe failed: Get "https://10.132.0.47:8444/healthz": dial tcp 10.132.0.47:8444: connect: connection refused 98m Warning FailedMount pod/ogx-k8s-operator-controller-manager-59777cdbbb-j2w7b MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 81m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000630000, 1000639999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 81m Warning Unhealthy pod/kserve-controller-manager-7f999b5d64-6w2t8 Readiness probe failed: Get "http://10.133.0.60:8081/readyz": dial tcp 10.133.0.60:8081: connect: connection refused 31m Warning FailedToRetrieveImagePullSecret pod/spark-operator-webhook-5d466b7d66-bjfms Unable to retrieve some image pull secrets (spark-operator-webhook-dockercfg-ncltg); attempting to pull the image may not succeed. 31m Warning Unhealthy pod/rhods-dashboard-78d9976d4f-vpn4k Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 31m Warning Unhealthy pod/rhods-dashboard-78d9976d4f-vpn4k Readiness probe failed: Get "https://10.133.0.58:8943/healthcheck": dial tcp 10.133.0.58:8943: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-78d9976d4f-lcrbf Readiness probe failed: Get "https://10.132.0.51:8943/healthcheck": dial tcp 10.132.0.51:8943: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-78d9976d4f-lcrbf Readiness probe failed: Get "https://10.132.0.51:8444/healthz": dial tcp 10.132.0.51:8444: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-78d9976d4f-lcrbf Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 31m Warning Unhealthy pod/rhods-dashboard-78d9976d4f-vpn4k Readiness probe failed: Get "https://10.133.0.58:8444/healthz": dial tcp 10.133.0.58:8444: connect: connection refused 31m Warning FailedMount pod/automl-ui-74d9b97d49-x68qc MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 31m Warning FailedMount pod/agent-ops-ui-7f9fbf68-lq7cv MountVolume.SetUp failed for volume "proxy-tls" : secret "agent-ops-proxy-tls" not found 31m Warning FailedMount pod/maas-ui-74675d48f9-wjrcd MountVolume.SetUp failed for volume "proxy-tls" : secret "maas-proxy-tls" not found 30m Warning FailedMount pod/maas-controller-56c898799-m5mvz MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found 30m Warning FailedMount pod/ogx-k8s-operator-controller-manager-59777cdbbb-6gzpt MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 30m Warning Unhealthy pod/notebook-controller-deployment-75c8689979-2s8r8 Readiness probe failed: Get "http://10.133.0.109:8081/readyz": dial tcp 10.133.0.109:8081: connect: connection refused 30m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000630000, 1000639999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 30m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000630000, 1000639999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 30m Warning FailedMount pod/kuberay-operator-7d87f54798-l5rl9 MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 29m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-4k4st Readiness probe failed: Get "https://10.132.0.108:8943/healthcheck": dial tcp 10.132.0.108:8943: connect: connection refused 29m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-4k4st Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 29m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-4k4st Readiness probe failed: Get "https://10.132.0.108:8444/healthz": dial tcp 10.132.0.108:8444: connect: connection refused 29m Warning FailedCreate replicaset/spark-operator-webhook-5d466b7d66 Error creating: pods "spark-operator-webhook-5d466b7d66-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-webhook: serviceaccount "spark-operator-webhook" not found 29m Warning Unhealthy pod/rhods-dashboard-7b658c686f-wn6p5 Readiness probe failed: Get "https://10.133.0.118:8444/healthz": dial tcp 10.133.0.118:8444: connect: connection refused 29m Warning Unhealthy pod/rhods-dashboard-7b658c686f-wn6p5 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 29m Warning Unhealthy pod/rhods-dashboard-7b658c686f-wn6p5 Readiness probe failed: Get "https://10.133.0.118:8943/healthcheck": dial tcp 10.133.0.118:8943: connect: connection refused 28m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-9c6vt Readiness probe failed: Get "https://10.133.0.110:8444/healthz": dial tcp 10.133.0.110:8444: connect: connection refused 28m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-9c6vt Readiness probe failed: Get "https://10.133.0.110:8943/healthcheck": dial tcp 10.133.0.110:8943: connect: connection refused 28m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-9c6vt Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 28m Warning Unhealthy pod/rhods-dashboard-7b658c686f-29gd9 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 28m Warning Unhealthy pod/rhods-dashboard-7b658c686f-29gd9 Readiness probe errored and resulted in unknown state: rpc error: code = NotFound desc = container is not created or running: checking if PID of 9848a498f027fd54faa948eaa3cedbd52fabfe7d242313a6e89ba80665671bfb is running failed: container process not found 28m Warning Unhealthy pod/rhods-dashboard-7b658c686f-29gd9 Readiness probe failed: Get "https://10.132.0.115:8943/healthcheck": dial tcp 10.132.0.115:8943: connect: connection refused 28m Warning Unhealthy pod/rhods-dashboard-7b658c686f-29gd9 Readiness probe failed: Get "https://10.132.0.115:8444/healthz": dial tcp 10.132.0.115:8444: connect: connection refused 23m Warning FailedMount pod/aihub-controller-manager-d877888b8-tgqn2 MountVolume.SetUp failed for volume "metrics-certs" : secret "aihub-controller-manager-metrics-tls" not found 23m Warning FailedMount pod/model-registry-ui-7897899d9b-cqhxs MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 23m Warning FailedToUpdateEndpoint endpoints/aihub-controller-manager-metrics-service Failed to update endpoint redhat-ods-applications/aihub-controller-manager-metrics-service: Operation cannot be fulfilled on endpoints "aihub-controller-manager-metrics-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/aihub-controller-manager-metrics-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: f637f8af-8edb-4267-8e7b-baa6d267a082, UID in object meta: 22m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-lvzfm Readiness probe failed: Get "https://10.132.0.123:8943/healthcheck": dial tcp 10.132.0.123:8943: connect: connection refused 22m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-lvzfm Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-09-06T18:30:07Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.21.30 Kubernetes Version: v1.34.9 --- Namespaces --- namespace/opendatahub namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy ResolutionFailed: status=True reason=ErrorPreventedResolution message=error using catalogsource openshift-marketplace/rhoai-catalog-dev: failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF" --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.2 Authorino Operator 1.4.2 authorino-operator.v1.4.1 Installing cert-manager-operator.v1.20.0 cert-manager Operator for Red Hat OpenShift 1.20.0 cert-manager-operator.v1.19.0 Installing custom-metrics-autoscaler.v2.19.0-3 Custom Metrics Autoscaler 2.19.0-3 custom-metrics-autoscaler.v2.19.0-2 Installing dns-operator.v1.4.1 DNS Operator 1.4.1 dns-operator.v1.4.0 Installing kueue-operator.v1.4.1 Red Hat build of Kueue 1.4.1 Installing limitador-operator.v1.4.1 Limitador Operator 1.4.1 limitador-operator.v1.4.0 Installing rhcl-operator.v1.4.2 Red Hat Connectivity Link 1.4.2 rhcl-operator.v1.4.1 Installing rhods-operator.3.6.0-ea.1 Red Hat OpenShift AI 3.6.0-ea.1 Installing --- CSVs not in Succeeded phase --- authorino-operator.v1.4.2: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace cert-manager-operator.v1.20.0: phase=Installing reason=Copied message=The operator is running in cert-manager-operator but is managing this namespace custom-metrics-autoscaler.v2.19.0-3: phase=Installing reason=Copied message=The operator is running in openshift-keda but is managing this namespace dns-operator.v1.4.1: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace kueue-operator.v1.4.1: phase=Installing reason=Copied message=The operator is running in openshift-kueue-operator but is managing this namespace limitador-operator.v1.4.1: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace rhcl-operator.v1.4.2: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace rhods-operator.3.6.0-ea.1: phase=Installing reason=InstallWaiting message=installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aigateway: managementState=Managed aipipelines: managementState=Managed batchGateway: managementState=Removed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Removed llamastackoperator: managementState=n/a mcplifecycleoperator: managementState=Removed mlflowoperator: managementState=Managed modelregistry: managementState=Managed modelsAsAService: managementState=Managed ogx: managementState=Managed ray: managementState=Managed sparkoperator: managementState=Managed trainer: managementState=Managed trainingoperator: managementState=Removed trustyai: managementState=Managed workbenches: managementState=Managed workbenchesV2: managementState=Removed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= ModulesReady: status=True reason= message= AIGatewayReady: status=True reason= message= AIPipelinesReady: status=True reason= message= BatchGatewayReady: status=False reason=Removed message=Submodule ManagementState is set to Removed DashboardReady: status=True reason=AllDependentsHealthy message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=False reason=Removed message=Component ManagementState is set to Removed MCPLifecycleOperatorReady: status=False reason=Removed message=Module ManagementState is set to Removed MLflowOperatorReady: status=True reason=Ready message=MLflowOperator is ready to manage MLflow custom resources ModelRegistryReady: status=True reason=AllDependentsHealthy message= ModelsAsAServiceReady: status=True reason=Ready message=modelsAsAService is Managed and deployments are available OGXReady: status=True reason=AllDependentsHealthy message= RayReady: status=True reason= message= SparkOperatorReady: status=True reason=AllDependentsHealthy message= TrainerReady: status=True reason= message= TrustyAIReady: status=True reason= message= WorkbenchesReady: status=True reason=ReconcileSuccess message=Workbenches component is ready WorkbenchesV2Ready: status=False reason=Removed message=Submodule ManagementState is set to Removed --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 117m Ready 2026-09-06T16:32:30Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=False reason=Removed message=Monitoring is not enabled MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in Monitoring CR OpenTelemetryCollectorAvailable: status=False reason=MetricsAndTracesNotConfigured message=Metrics and traces are not configured in Monitoring CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in Monitoring CR NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR PersesAvailable: status=False reason=MetricsAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured ProvisioningSucceeded: status=True reason=Available message= Ready: status=True reason=Available message= --- Component CRs (Ready) --- NAME READY REASON VERSION aigateway.components.platform.opendatahub.io/default-aigateway True 1.26.2 NAME APPNAMESPACE AGE aihub.components.platform.opendatahub.io/default-aihub redhat-ods-applications 11m NAME PHASE READY REASON URL AGE dashboard.components.platform.opendatahub.io/default-dashboard Ready True AllDependentsHealthy https://rh-ai.apps.122c55ac2b1d4d0d2a11.prod.konflux-ocp-ci.dev/ 31m NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON mlflowoperator.components.platform.opendatahub.io/default-mlflowoperator True Ready NAME READY REASON DISTRIBUTION ogx.components.platform.opendatahub.io/default-ogx True AllDependentsHealthy Red Hat OpenShift AI NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON sparkoperator.components.platform.opendatahub.io/default-sparkoperator True AllDependentsHealthy NAME READY REASON AGE trainer.components.platform.opendatahub.io/default-trainer True 17m NAME READY REASON trustyai.components.platform.opendatahub.io/default-trustyai True NAME READY REASON PHASE workbenches.components.platform.opendatahub.io/default-workbenches True ReconcileSuccess Ready --- Deployments not fully ready (ODH/RHOAI namespaces) --- redhat-ods-applications/agent-ops-ui: ready=0/1 available=0 redhat-ods-applications/ai-gateway-operator: ready=0/1 available=0 redhat-ods-applications/aihub-controller-manager: ready=0/1 available=0 redhat-ods-applications/automl-ui: ready=0/1 available=0 redhat-ods-applications/autorag-ui: ready=0/1 available=0 redhat-ods-applications/catalog-controller-manager: ready=0/1 available=0 redhat-ods-applications/dashboard-operator: ready=0/2 available=0 redhat-ods-applications/dashboard-redirect: ready=0/2 available=0 redhat-ods-applications/data-science-pipelines-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/eval-hub-ui: ready=0/1 available=0 redhat-ods-applications/feast-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/gen-ai-ui: ready=0/1 available=0 redhat-ods-applications/kserve-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-localmodel-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-trainer-controller-manager: ready=0/1 available=0 redhat-ods-applications/kuberay-operator: ready=0/1 available=0 redhat-ods-applications/llmisvc-controller-manager: ready=0/1 available=0 redhat-ods-applications/maas-controller: ready=0/1 available=0 redhat-ods-applications/maas-ui: ready=0/1 available=0 redhat-ods-applications/mlflow: ready=0/1 available=0 redhat-ods-applications/mlflow-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/mlflow-ui: ready=0/1 available=0 redhat-ods-applications/model-registry-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-registry-ui: ready=0/1 available=0 redhat-ods-applications/model-serving-api: ready=0/1 available=0 redhat-ods-applications/notebook-controller-deployment: ready=0/1 available=0 redhat-ods-applications/notebooks-ui: ready=0/1 available=0 redhat-ods-applications/odh-model-controller: ready=0/1 available=0 redhat-ods-applications/odh-notebook-controller-manager: ready=0/1 available=0 redhat-ods-applications/ogx-k8s-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/opendatahub-feast-operator: ready=0/1 available=0 redhat-ods-applications/opendatahub-ogx-operator: ready=0/1 available=0 redhat-ods-applications/postgres-deployment: ready=0/1 available=0 redhat-ods-applications/rhods-dashboard: ready=0/2 available=0 redhat-ods-applications/spark-operator-controller: ready=0/1 available=0 redhat-ods-applications/spark-operator-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook: ready=0/1 available=0 redhat-ods-applications/trainer-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/trustyai-service-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/workbenches-operator: ready=0/1 available=0 redhat-ods-operator/rhods-operator: ready=0/3 available=0 rhoai-model-registries/model-catalog: ready=0/1 available=0 rhoai-model-registries/model-catalog-postgres: ready=0/1 available=0 --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- redhat-ods-applications/agent-ops-ui-7f9fbf68-msrj6: phase=Pending reason= redhat-ods-applications/aihub-controller-manager-d877888b8-wsx4s: phase=Pending reason= redhat-ods-applications/catalog-controller-manager-76985cd459-dnvc8: phase=Pending reason= redhat-ods-applications/dashboard-operator-669d6564bd-r5nqx: phase=Pending reason= redhat-ods-applications/dashboard-operator-669d6564bd-z5xvv: phase=Pending reason= redhat-ods-applications/dashboard-redirect-76d74cfc9f-fznfl: phase=Pending reason= redhat-ods-applications/data-science-pipelines-operator-controller-manager-56b8f89khdkb: phase=Pending reason= redhat-ods-applications/feast-operator-controller-manager-5fc6bb8d67-nb66k: phase=Pending reason= redhat-ods-applications/gen-ai-ui-7c7dbc9d78-9dh9f: phase=Pending reason= redhat-ods-applications/kserve-controller-manager-5fbbd6dd88-b729h: phase=Pending reason= redhat-ods-applications/kserve-localmodel-controller-manager-fc8c879d7-fknxk: phase=Pending reason= redhat-ods-applications/kserve-module-controller-manager-66ff7c977-nhf2c: phase=Pending reason= redhat-ods-applications/kubeflow-trainer-controller-manager-5449979c85-cnhbk: phase=Pending reason= redhat-ods-applications/llmisvc-controller-manager-6df7b6c5d4-qksnx: phase=Pending reason= redhat-ods-applications/maas-controller-56c898799-x8jbb: phase=Pending reason= redhat-ods-applications/maas-ui-74675d48f9-cxmkg: phase=Pending reason= redhat-ods-applications/mlflow-operator-controller-manager-5bbb468f5f-btsd8: phase=Pending reason= redhat-ods-applications/mlflow-ui-7bb6d46f5f-wd6r5: phase=Pending reason= redhat-ods-applications/model-registry-operator-controller-manager-758fcb8c6b-qjrvp: phase=Pending reason= redhat-ods-applications/model-serving-api-7df8c9655f-2qdks: phase=Pending reason= redhat-ods-applications/notebook-controller-deployment-75c8689979-6fn72: phase=Pending reason= redhat-ods-applications/notebooks-ui-6577cc467-n78h6: phase=Pending reason= redhat-ods-applications/odh-model-controller-6b9dd8b59d-gqxm5: phase=Pending reason= redhat-ods-applications/ogx-k8s-operator-controller-manager-59777cdbbb-ncl4z: phase=Pending reason= redhat-ods-applications/rhods-dashboard-78d9976d4f-f8w8s: phase=Pending reason= redhat-ods-applications/rhods-dashboard-78d9976d4f-jd7mm: phase=Pending reason= redhat-ods-applications/spark-operator-controller-c594b967f-ffppx: phase=Pending reason= redhat-ods-applications/spark-operator-webhook-5d466b7d66-jpq9x: phase=Pending reason= redhat-ods-applications/trainer-operator-controller-manager-6b585cbb9f-dq7fg: phase=Pending reason= redhat-ods-operator/rhods-operator-7897f9b485-6cfmt: phase=Pending reason= redhat-ods-operator/rhods-operator-7897f9b485-fz5gz: phase=Pending reason= rhoai-model-registries/model-catalog-dbb95775b-scfvc: phase=Pending reason= rhoai-model-registries/model-catalog-postgres-7f94f4df48-4kr6j: phase=Pending reason= --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/opendatahub --- (no events) --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 10m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 118s Warning FailedScheduling pod/trainer-operator-controller-manager-6b585cbb9f-dq7fg no nodes available to schedule pods 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-fknxk no nodes available to schedule pods 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 118s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-fknxk no nodes available to schedule pods 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m17s Warning FailedScheduling pod/kserve-localmodel-controller-manager-fc8c879d7-fknxk 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/mlflow-ui-7bb6d46f5f-wd6r5 no nodes available to schedule pods 7m17s Warning FailedScheduling pod/agent-ops-ui-7f9fbf68-msrj6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/agent-ops-ui-7f9fbf68-msrj6 no nodes available to schedule pods 113s Warning FailedScheduling pod/agent-ops-ui-7f9fbf68-msrj6 no nodes available to schedule pods 67m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m16s Warning FailedScheduling pod/mlflow-ui-7bb6d46f5f-wd6r5 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 74m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m17s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-gqxm5 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-gqxm5 no nodes available to schedule pods 113s Warning FailedScheduling pod/odh-model-controller-6b9dd8b59d-gqxm5 no nodes available to schedule pods 82m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114s Warning FailedScheduling pod/notebooks-ui-6577cc467-n78h6 no nodes available to schedule pods 118s Warning FailedScheduling pod/notebooks-ui-6577cc467-n78h6 no nodes available to schedule pods 7m17s Warning FailedScheduling pod/notebooks-ui-6577cc467-n78h6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m24s Warning FailedScheduling pod/maas-controller-56c898799-x8jbb no nodes available to schedule pods 2m28s Warning FailedScheduling pod/maas-controller-56c898799-x8jbb no nodes available to schedule pods 7m41s Warning FailedScheduling pod/maas-controller-56c898799-x8jbb 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m17s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-wsx4s 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-wsx4s no nodes available to schedule pods 113s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-wsx4s no nodes available to schedule pods 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 10m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m57s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 21m Warning FailedScheduling pod/kserve-localmodelnode-agent-927tf 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 21m Warning FailedScheduling pod/kserve-localmodelnode-agent-927tf 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 83m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-6fn72 no nodes available to schedule pods 118s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-6fn72 no nodes available to schedule pods 7m18s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-6fn72 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 113s Warning FailedScheduling pod/llmisvc-controller-manager-6df7b6c5d4-qksnx no nodes available to schedule pods 118s Warning FailedScheduling pod/llmisvc-controller-manager-6df7b6c5d4-qksnx no nodes available to schedule pods 7m16s Warning FailedScheduling pod/llmisvc-controller-manager-6df7b6c5d4-qksnx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 31m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 29m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 113s Warning FailedScheduling pod/trainer-operator-controller-manager-6b585cbb9f-dq7fg no nodes available to schedule pods 7m16s Warning FailedScheduling pod/trainer-operator-controller-manager-6b585cbb9f-dq7fg 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 30m Warning FailedScheduling pod/kserve-localmodelnode-agent-mmmpl 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m39s Warning FailedScheduling pod/catalog-controller-manager-76985cd459-dnvc8 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2m28s Warning FailedScheduling pod/catalog-controller-manager-76985cd459-dnvc8 no nodes available to schedule pods 2m23s Warning FailedScheduling pod/catalog-controller-manager-76985cd459-dnvc8 no nodes available to schedule pods 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 22m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 22m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 21m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m24s Warning FailedScheduling pod/spark-operator-webhook-5d466b7d66-jpq9x no nodes available to schedule pods 2m28s Warning FailedScheduling pod/spark-operator-webhook-5d466b7d66-jpq9x no nodes available to schedule pods 7m40s Warning FailedScheduling pod/spark-operator-webhook-5d466b7d66-jpq9x 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 17m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 17m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m39s Warning FailedScheduling pod/postgres-deployment-86754cb759-4zt6s 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 30m Warning FailedScheduling pod/kserve-localmodelnode-agent-nh54p 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 60m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 84m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 22m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 21m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 10m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m57s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 21m Warning FailedScheduling pod/kserve-localmodelnode-agent-nmql4 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 74m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m18s Warning FailedScheduling pod/dashboard-operator-669d6564bd-r5nqx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/dashboard-operator-669d6564bd-r5nqx no nodes available to schedule pods 114s Warning FailedScheduling pod/dashboard-operator-669d6564bd-r5nqx no nodes available to schedule pods 7m39s Warning FailedScheduling pod/dashboard-operator-669d6564bd-z5xvv 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2m28s Warning FailedScheduling pod/dashboard-operator-669d6564bd-z5xvv no nodes available to schedule pods 2m23s Warning FailedScheduling pod/dashboard-operator-669d6564bd-z5xvv no nodes available to schedule pods 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m17s Warning FailedScheduling pod/rhods-dashboard-78d9976d4f-f8w8s 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7m40s Warning FailedScheduling pod/model-registry-operator-controller-manager-758fcb8c6b-qjrvp 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m28s Warning FailedScheduling pod/model-registry-operator-controller-manager-758fcb8c6b-qjrvp no nodes available to schedule pods 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m24s Warning FailedScheduling pod/model-registry-operator-controller-manager-758fcb8c6b-qjrvp no nodes available to schedule pods 21m Warning FailedScheduling pod/kserve-localmodelnode-agent-nmql4 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 13m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 118s Warning FailedScheduling pod/rhods-dashboard-78d9976d4f-f8w8s no nodes available to schedule pods 114s Warning FailedScheduling pod/rhods-dashboard-78d9976d4f-f8w8s no nodes available to schedule pods 57m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 49m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m39s Warning FailedScheduling pod/dashboard-redirect-76d74cfc9f-fznfl 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2m28s Warning FailedScheduling pod/dashboard-redirect-76d74cfc9f-fznfl no nodes available to schedule pods 2m23s Warning FailedScheduling pod/dashboard-redirect-76d74cfc9f-fznfl no nodes available to schedule pods 7m40s Warning FailedScheduling pod/rhods-dashboard-78d9976d4f-jd7mm 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 11m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning FailedScheduling pod/kserve-localmodelnode-agent-rw779 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 47m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m28s Warning FailedScheduling pod/rhods-dashboard-78d9976d4f-jd7mm no nodes available to schedule pods 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m24s Warning FailedScheduling pod/rhods-dashboard-78d9976d4f-jd7mm no nodes available to schedule pods 30m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 30m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 29m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 24m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning FailedScheduling pod/kserve-localmodelnode-agent-s8gsw 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 21m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 10m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m17s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f89khdkb 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f89khdkb no nodes available to schedule pods 114s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f89khdkb no nodes available to schedule pods 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 16m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 15m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114s Warning FailedScheduling pod/model-serving-api-7df8c9655f-2qdks no nodes available to schedule pods 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m57s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 118s Warning FailedScheduling pod/model-serving-api-7df8c9655f-2qdks no nodes available to schedule pods 7m17s Warning FailedScheduling pod/model-serving-api-7df8c9655f-2qdks 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 10m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m57s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m18s Warning FailedScheduling pod/maas-ui-74675d48f9-cxmkg 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/maas-ui-74675d48f9-cxmkg no nodes available to schedule pods 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 103m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m39s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-ncl4z 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114s Warning FailedScheduling pod/maas-ui-74675d48f9-cxmkg no nodes available to schedule pods 102m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-cnhbk no nodes available to schedule pods 118s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-cnhbk no nodes available to schedule pods 7m18s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-cnhbk 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 84m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m40s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-nb66k 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2m28s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-nb66k no nodes available to schedule pods 2m24s Warning FailedScheduling pod/feast-operator-controller-manager-5fc6bb8d67-nb66k no nodes available to schedule pods 2m24s Warning FailedScheduling pod/spark-operator-controller-c594b967f-ffppx no nodes available to schedule pods 2m28s Warning FailedScheduling pod/spark-operator-controller-c594b967f-ffppx no nodes available to schedule pods 7m40s Warning FailedScheduling pod/spark-operator-controller-c594b967f-ffppx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 31m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/mcp-lifecycle-operator-controller-manager-metrics-monitor "redhat-ods-applications/mcp-lifecycle-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 83m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m57s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 113s Warning FailedScheduling pod/mlflow-ui-7bb6d46f5f-wd6r5 no nodes available to schedule pods 82m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 11m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 23m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 31m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 47m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m39s Warning FailedScheduling pod/mlflow-6f9ddc956d-6m4z9 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 49m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m18s Warning FailedScheduling pod/gen-ai-ui-7c7dbc9d78-9dh9f 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/gen-ai-ui-7c7dbc9d78-9dh9f no nodes available to schedule pods 114s Warning FailedScheduling pod/gen-ai-ui-7c7dbc9d78-9dh9f no nodes available to schedule pods 113s Warning FailedScheduling pod/kserve-module-controller-manager-66ff7c977-nhf2c no nodes available to schedule pods 118s Warning FailedScheduling pod/kserve-module-controller-manager-66ff7c977-nhf2c no nodes available to schedule pods 7m17s Warning FailedScheduling pod/kserve-module-controller-manager-66ff7c977-nhf2c 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 57m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 57m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m23s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-ncl4z no nodes available to schedule pods 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 67m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 2m28s Warning FailedScheduling pod/ogx-k8s-operator-controller-manager-59777cdbbb-ncl4z no nodes available to schedule pods 7m18s Warning FailedScheduling pod/mlflow-operator-controller-manager-5bbb468f5f-btsd8 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/mlflow-operator-controller-manager-5bbb468f5f-btsd8 no nodes available to schedule pods 114s Warning FailedScheduling pod/mlflow-operator-controller-manager-5bbb468f5f-btsd8 no nodes available to schedule pods 7m17s Warning FailedScheduling pod/kserve-controller-manager-5fbbd6dd88-b729h 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 118s Warning FailedScheduling pod/kserve-controller-manager-5fbbd6dd88-b729h no nodes available to schedule pods 113s Warning FailedScheduling pod/kserve-controller-manager-5fbbd6dd88-b729h no nodes available to schedule pods 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 72m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 74m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 82m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 117m Warning FailedMount pod/odh-observability-59db79cb46-8nd4t MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 117m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 117m Warning FailedMount pod/kuberay-operator-7d87f54798-dkgpd MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 117m Warning FailedMount pod/model-registry-operator-controller-manager-758fcb8c6b-c9wvs MountVolume.SetUp failed for volume "server-cert" : secret "model-registry-operator-controller-manager-metrics-service" not found 117m Warning FailedMount pod/model-registry-operator-controller-manager-758fcb8c6b-c9wvs MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 117m Warning FailedMount pod/mlflow-operator-controller-manager-5bbb468f5f-8ww9r MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 117m Warning FailedMount pod/notebooks-ui-6577cc467-55hxj MountVolume.SetUp failed for volume "proxy-tls" : secret "notebooks-proxy-tls" not found 117m Warning FailedMount pod/eval-hub-ui-869b89d9b4-x7vxb MountVolume.SetUp failed for volume "proxy-tls" : secret "eval-hub-proxy-tls" not found 117m Warning FailedMount pod/model-registry-ui-7897899d9b-ls82g MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 117m Warning FailedMount pod/odh-notebook-controller-manager-76dd46fbc-zl85r MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 117m Warning FailedMount pod/mlflow-ui-7bb6d46f5f-l56tj MountVolume.SetUp failed for volume "proxy-tls" : secret "mlflow-proxy-tls" not found 115m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-xkqfs Readiness probe failed: Get "https://10.133.0.55:8943/healthcheck": dial tcp 10.133.0.55:8943: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-xkqfs Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 115m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-xkqfs Readiness probe failed: Get "https://10.133.0.55:8444/healthz": dial tcp 10.133.0.55:8444: connect: connection refused 115m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-x2vhh Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 115m Warning FailedMount pod/maas-controller-56c898799-jxzbq MountVolume.SetUp failed for volume "metrics-certs" : secret "maas-controller-metrics-tls" not found 115m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-x2vhh Readiness probe failed: Get "https://10.132.0.47:8943/healthcheck": dial tcp 10.132.0.47:8943: connect: connection refused 115m Warning FailedMount pod/maas-controller-56c898799-jxzbq MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found 115m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-x2vhh Readiness probe failed: Get "https://10.132.0.47:8444/healthz": dial tcp 10.132.0.47:8444: connect: connection refused 98m Warning FailedMount pod/ogx-k8s-operator-controller-manager-59777cdbbb-j2w7b MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 81m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "run-as-ray-user": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000630000, 1000639999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 81m Warning Unhealthy pod/kserve-controller-manager-7f999b5d64-6w2t8 Readiness probe failed: Get "http://10.133.0.60:8081/readyz": dial tcp 10.133.0.60:8081: connect: connection refused 31m Warning FailedToRetrieveImagePullSecret pod/spark-operator-webhook-5d466b7d66-bjfms Unable to retrieve some image pull secrets (spark-operator-webhook-dockercfg-ncltg); attempting to pull the image may not succeed. 31m Warning Unhealthy pod/rhods-dashboard-78d9976d4f-vpn4k Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 31m Warning Unhealthy pod/rhods-dashboard-78d9976d4f-vpn4k Readiness probe failed: Get "https://10.133.0.58:8943/healthcheck": dial tcp 10.133.0.58:8943: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-78d9976d4f-lcrbf Readiness probe failed: Get "https://10.132.0.51:8943/healthcheck": dial tcp 10.132.0.51:8943: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-78d9976d4f-lcrbf Readiness probe failed: Get "https://10.132.0.51:8444/healthz": dial tcp 10.132.0.51:8444: connect: connection refused 31m Warning Unhealthy pod/rhods-dashboard-78d9976d4f-lcrbf Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 31m Warning Unhealthy pod/rhods-dashboard-78d9976d4f-vpn4k Readiness probe failed: Get "https://10.133.0.58:8444/healthz": dial tcp 10.133.0.58:8444: connect: connection refused 31m Warning FailedMount pod/automl-ui-74d9b97d49-x68qc MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 31m Warning FailedMount pod/agent-ops-ui-7f9fbf68-lq7cv MountVolume.SetUp failed for volume "proxy-tls" : secret "agent-ops-proxy-tls" not found 31m Warning FailedMount pod/maas-ui-74675d48f9-wjrcd MountVolume.SetUp failed for volume "proxy-tls" : secret "maas-proxy-tls" not found 30m Warning FailedMount pod/maas-controller-56c898799-m5mvz MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found 30m Warning FailedMount pod/ogx-k8s-operator-controller-manager-59777cdbbb-6gzpt MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 30m Warning Unhealthy pod/notebook-controller-deployment-75c8689979-2s8r8 Readiness probe failed: Get "http://10.133.0.109:8081/readyz": dial tcp 10.133.0.109:8081: connect: connection refused 30m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000630000, 1000639999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 30m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000630000, 1000639999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 30m Warning FailedMount pod/kuberay-operator-7d87f54798-l5rl9 MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 29m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-4k4st Readiness probe failed: Get "https://10.132.0.108:8943/healthcheck": dial tcp 10.132.0.108:8943: connect: connection refused 29m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-4k4st Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 29m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-4k4st Readiness probe failed: Get "https://10.132.0.108:8444/healthz": dial tcp 10.132.0.108:8444: connect: connection refused 29m Warning FailedCreate replicaset/spark-operator-webhook-5d466b7d66 Error creating: pods "spark-operator-webhook-5d466b7d66-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-webhook: serviceaccount "spark-operator-webhook" not found 29m Warning Unhealthy pod/rhods-dashboard-7b658c686f-wn6p5 Readiness probe failed: Get "https://10.133.0.118:8444/healthz": dial tcp 10.133.0.118:8444: connect: connection refused 29m Warning Unhealthy pod/rhods-dashboard-7b658c686f-wn6p5 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 29m Warning Unhealthy pod/rhods-dashboard-7b658c686f-wn6p5 Readiness probe failed: Get "https://10.133.0.118:8943/healthcheck": dial tcp 10.133.0.118:8943: connect: connection refused 28m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-9c6vt Readiness probe failed: Get "https://10.133.0.110:8444/healthz": dial tcp 10.133.0.110:8444: connect: connection refused 28m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-9c6vt Readiness probe failed: Get "https://10.133.0.110:8943/healthcheck": dial tcp 10.133.0.110:8943: connect: connection refused 28m Warning Unhealthy pod/rhods-dashboard-797b4b4bd7-9c6vt Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 28m Warning Unhealthy pod/rhods-dashboard-7b658c686f-29gd9 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 28m Warning Unhealthy pod/rhods-dashboard-7b658c686f-29gd9 Readiness probe errored and resulted in unknown state: rpc error: code = NotFound desc = container is not created or running: checking if PID of 9848a498f027fd54faa948eaa3cedbd52fabfe7d242313a6e89ba80665671bfb is running failed: container process not found 28m Warning Unhealthy pod/rhods-dashboard-7b658c686f-29gd9 Readiness probe failed: Get "https://10.132.0.115:8943/healthcheck": dial tcp 10.132.0.115:8943: connect: connection refused 28m Warning Unhealthy pod/rhods-dashboard-7b658c686f-29gd9 Readiness probe failed: Get "https://10.132.0.115:8444/healthz": dial tcp 10.132.0.115:8444: connect: connection refused 23m Warning FailedMount pod/aihub-controller-manager-d877888b8-tgqn2 MountVolume.SetUp failed for volume "metrics-certs" : secret "aihub-controller-manager-metrics-tls" not found 23m Warning FailedMount pod/model-registry-ui-7897899d9b-cqhxs MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 23m Warning FailedToUpdateEndpoint endpoints/aihub-controller-manager-metrics-service Failed to update endpoint redhat-ods-applications/aihub-controller-manager-metrics-service: Operation cannot be fulfilled on endpoints "aihub-controller-manager-metrics-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/aihub-controller-manager-metrics-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: f637f8af-8edb-4267-8e7b-baa6d267a082, UID in object meta: 22m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-lvzfm Readiness probe failed: Get "https://10.132.0.123:8943/healthcheck": dial tcp 10.132.0.123:8943: connect: connection refused 22m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-lvzfm Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 22m Warning FailedToUpdateEndpoint endpoints/odh-dashboard-model-registry-ui Failed to update endpoint redhat-ods-applications/odh-dashboard-model-registry-ui: Operation cannot be fulfilled on endpoints "odh-dashboard-model-registry-ui": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/odh-dashboard-model-registry-ui, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d6e760d3-c786-4ad9-b390-a8662e48c9e1, UID in object meta: 22m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-lvzfm Readiness probe failed: Get "https://10.132.0.123:8444/healthz": dial tcp 10.132.0.123:8444: connect: connection refused 22m Warning Unhealthy pod/rhods-dashboard-6f777fd8f6-9hxbh Readiness probe failed: Get "https://10.133.0.124:8444/healthz": dial tcp 10.133.0.124:8444: connect: connection refused 22m Warning Unhealthy pod/rhods-dashboard-6f777fd8f6-9hxbh Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 22m Warning Unhealthy pod/rhods-dashboard-6f777fd8f6-9hxbh Readiness probe failed: Get "https://10.133.0.124:8943/healthcheck": dial tcp 10.133.0.124:8943: connect: connection refused 22m Warning Unhealthy pod/rhods-dashboard-6f777fd8f6-tmfjz Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 22m Warning Unhealthy pod/rhods-dashboard-6f777fd8f6-tmfjz Readiness probe failed: Get "https://10.132.0.134:8943/healthcheck": dial tcp 10.132.0.134:8943: connect: connection refused 21m Warning Unhealthy pod/rhods-dashboard-6f777fd8f6-tmfjz Readiness probe failed: Get "https://10.132.0.134:8444/healthz": dial tcp 10.132.0.134:8444: connect: connection refused 21m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000630000, 1000639999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 21m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000630000, 1000639999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 21m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-8b4rk Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 21m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-8b4rk Readiness probe failed: Get "https://10.132.0.140:8943/healthcheck": dial tcp 10.132.0.140:8943: connect: connection refused 21m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-8b4rk Readiness probe failed: Get "https://10.132.0.140:8444/healthz": dial tcp 10.132.0.140:8444: connect: connection refused 20m Warning FailedToUpdateEndpoint endpoints/odh-dashboard-mlflow-ui Failed to update endpoint redhat-ods-applications/odh-dashboard-mlflow-ui: Operation cannot be fulfilled on endpoints "odh-dashboard-mlflow-ui": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/odh-dashboard-mlflow-ui, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 78912c62-dbdc-47fa-b621-c4d897fab337, UID in object meta: 20m Warning FailedMount pod/trustyai-service-operator-controller-manager-d6bf47757-lltjx MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 20m Warning Unhealthy pod/rhods-dashboard-c56bdc4b5-sphjv Readiness probe failed: Get "https://10.133.0.126:8444/healthz": dial tcp 10.133.0.126:8444: connect: connection refused 20m Warning Unhealthy pod/rhods-dashboard-c56bdc4b5-sphjv Readiness probe failed: Get "https://10.133.0.126:8943/healthcheck": dial tcp 10.133.0.126:8943: connect: connection refused 20m Warning Unhealthy pod/rhods-dashboard-c56bdc4b5-sphjv Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 20m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-nvrqx Readiness probe failed: Get "https://10.132.0.146:8444/healthz": dial tcp 10.132.0.146:8444: connect: connection refused 20m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-nvrqx Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 20m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-nvrqx Readiness probe failed: Get "https://10.132.0.146:8943/healthcheck": dial tcp 10.132.0.146:8943: connect: connection refused 19m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-wn46m Readiness probe failed: Get "https://10.133.0.122:8943/healthcheck": dial tcp 10.133.0.122:8943: connect: connection refused 19m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-wn46m Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 19m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-wn46m Readiness probe failed: Get "https://10.133.0.122:8444/healthz": dial tcp 10.133.0.122:8444: connect: connection refused 19m Warning Unhealthy pod/rhods-dashboard-c56bdc4b5-r29hj Readiness probe failed: Get "https://10.132.0.142:8943/healthcheck": dial tcp 10.132.0.142:8943: connect: connection refused 19m Warning Unhealthy pod/rhods-dashboard-c56bdc4b5-r29hj Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 18m Warning Unhealthy pod/rhods-dashboard-c56bdc4b5-r29hj Readiness probe failed: Get "https://10.132.0.142:8444/healthz": dial tcp 10.132.0.142:8444: connect: connection refused 16m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000630000, 1000639999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 16m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000630000, 1000639999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 16m Warning Unhealthy pod/rhods-dashboard-786954f754-4njsh Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 16m Warning Unhealthy pod/rhods-dashboard-786954f754-4njsh Readiness probe failed: Get "https://10.132.0.149:8943/healthcheck": dial tcp 10.132.0.149:8943: connect: connection refused 16m Warning Unhealthy pod/rhods-dashboard-786954f754-4njsh Readiness probe failed: Get "https://10.132.0.149:8444/healthz": dial tcp 10.132.0.149:8444: connect: connection refused 15m Warning Unhealthy pod/rhods-dashboard-786954f754-chlsw Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 15m Warning Unhealthy pod/rhods-dashboard-786954f754-chlsw Readiness probe failed: Get "https://10.133.0.129:8943/healthcheck": dial tcp 10.133.0.129:8943: connect: connection refused 15m Warning Unhealthy pod/rhods-dashboard-786954f754-chlsw Readiness probe failed: Get "https://10.133.0.129:8444/healthz": dial tcp 10.133.0.129:8444: connect: connection refused 14m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000630000, 1000639999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 14m Warning FailedCreate daemonset/kserve-localmodelnode-agent Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000630000, 1000639999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 14m Warning FailedMount pod/kserve-localmodel-controller-manager-fc8c879d7-x4s7n MountVolume.SetUp failed for volume "cert" : secret "localmodel-webhook-server-cert" not found 14m Warning FailedMount pod/odh-model-controller-57457c4c89-sqz4c MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 13m Warning FailedMount pod/mlflow-operator-controller-manager-5bbb468f5f-8x672 MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 12m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-2g7vz Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 12m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-2g7vz Readiness probe failed: Get "https://10.132.0.152:8943/healthcheck": dial tcp 10.132.0.152:8943: connect: connection refused 11m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-2g7vz Readiness probe failed: Get "https://10.132.0.152:8444/healthz": dial tcp 10.132.0.152:8444: connect: connection refused 11m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-nkl5c Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 11m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-nkl5c Readiness probe failed: Get "https://10.133.0.130:8943/healthcheck": dial tcp 10.133.0.130:8943: connect: connection refused 11m Warning FailedMount pod/eval-hub-ui-869b89d9b4-8t9wt MountVolume.SetUp failed for volume "proxy-tls" : secret "eval-hub-proxy-tls" not found 11m Warning FailedMount pod/model-registry-ui-7897899d9b-wp7hk MountVolume.SetUp failed for volume "proxy-tls" : secret "model-registry-proxy-tls" not found 11m Warning FailedMount pod/automl-ui-74d9b97d49-qmm28 MountVolume.SetUp failed for volume "proxy-tls" : secret "automl-proxy-tls" not found 11m Warning Unhealthy pod/rhods-dashboard-86c5fb98b4-nkl5c Readiness probe failed: Get "https://10.133.0.130:8444/healthz": dial tcp 10.133.0.130:8444: connect: connection refused 11m Warning FailedMount pod/model-registry-operator-controller-manager-758fcb8c6b-79m9p MountVolume.SetUp failed for volume "server-cert" : secret "model-registry-operator-controller-manager-metrics-service" not found 11m Warning FailedMount pod/maas-controller-56c898799-ckvmp MountVolume.SetUp failed for volume "metrics-certs" : secret "maas-controller-metrics-tls" not found 10m Warning Unhealthy pod/rhods-dashboard-c56bdc4b5-7bbtr Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 10m Warning Unhealthy pod/rhods-dashboard-c56bdc4b5-7bbtr Readiness probe failed: Get "https://10.132.0.169:8943/healthcheck": dial tcp 10.132.0.169:8943: connect: connection refused 10m Warning Unhealthy pod/rhods-dashboard-c56bdc4b5-7bbtr Readiness probe failed: Get "https://10.132.0.169:8444/healthz": dial tcp 10.132.0.169:8444: connect: connection refused 9m48s Warning Unhealthy pod/rhods-dashboard-c56bdc4b5-ktwm5 Readiness probe failed: Get "https://10.133.0.133:8943/healthcheck": dial tcp 10.133.0.133:8943: connect: connection refused 9m48s Warning Unhealthy pod/rhods-dashboard-c56bdc4b5-ktwm5 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 9m43s Warning Unhealthy pod/rhods-dashboard-c56bdc4b5-ktwm5 Readiness probe failed: Get "https://10.133.0.133:8444/healthz": dial tcp 10.133.0.133:8444: connect: connection refused 9m24s Warning FailedAttachVolume pod/mlflow-6f9ddc956d-tdvjh Multi-Attach error for volume "pvc-8e4d0327-02e8-49bd-8af4-7f0f1df7ea90" Volume is already exclusively attached to one node and can't be attached to another 9m21s Warning Unhealthy pod/notebook-controller-deployment-75c8689979-7vpmh Readiness probe failed: Get "http://10.132.0.174:8081/readyz": dial tcp 10.132.0.174:8081: connect: connection refused 8m35s Warning NodeNotReady pod/kubeflow-trainer-controller-manager-5449979c85-rc68t Node is not ready 8m35s Warning NodeNotReady pod/workbenches-operator-c674d9c9-9s28q Node is not ready 8m35s Warning NodeNotReady pod/kserve-localmodelnode-agent-rw779 Node is not ready 8m35s Warning NodeNotReady pod/data-science-pipelines-operator-controller-manager-56b8f89vf8sn Node is not ready 8m35s Warning NodeNotReady pod/agent-ops-ui-7f9fbf68-lq7cv Node is not ready 8m35s Warning NodeNotReady pod/ai-gateway-operator-89f69845b-wdnq6 Node is not ready 8m35s Warning NodeNotReady pod/kserve-controller-manager-5fbbd6dd88-5t9jr Node is not ready 8m35s Warning NodeNotReady pod/aihub-controller-manager-d877888b8-rz4dl Node is not ready 8m35s Warning NodeNotReady pod/rhods-dashboard-78d9976d4f-vpd2v Node is not ready 8m35s Warning NodeNotReady pod/model-registry-ui-7897899d9b-wp7hk Node is not ready 8m35s Warning NodeNotReady pod/model-serving-api-7df8c9655f-x97fd Node is not ready 8m35s Warning NodeNotReady pod/odh-model-controller-6b9dd8b59d-5plvv Node is not ready 8m35s Warning NodeNotReady pod/dashboard-operator-669d6564bd-zmng8 Node is not ready 8m35s Warning NodeNotReady pod/mlflow-ui-7bb6d46f5f-nqhgp Node is not ready 8m34s Warning NodeNotReady pod/automl-ui-74d9b97d49-qmm28 Node is not ready 8m34s Warning NodeNotReady pod/notebook-controller-deployment-75c8689979-7vpmh Node is not ready 8m34s Warning NodeNotReady pod/eval-hub-ui-869b89d9b4-8t9wt Node is not ready 8m34s Warning NodeNotReady pod/kserve-module-controller-manager-66ff7c977-cfw4g Node is not ready 8m34s Warning NodeNotReady pod/autorag-ui-58f548cbd-t4cxd Node is not ready 8m34s Warning NodeNotReady pod/kuberay-operator-7d87f54798-xrnbz Node is not ready 8m34s Warning NodeNotReady pod/trustyai-service-operator-controller-manager-d6bf47757-94jzl Node is not ready 8m34s Warning NodeNotReady pod/odh-notebook-controller-manager-76dd46fbc-c9n2l Node is not ready 8m34s Warning NodeNotReady pod/trainer-operator-controller-manager-6b585cbb9f-g4l4h Node is not ready 8m34s Warning NodeNotReady pod/gen-ai-ui-7c7dbc9d78-rwjvr Node is not ready 8m34s Warning NodeNotReady pod/opendatahub-ogx-operator-644d45d5b-nrx5b Node is not ready 8m34s Warning NodeNotReady pod/maas-ui-74675d48f9-wjrcd Node is not ready 8m34s Warning NodeNotReady pod/notebooks-ui-6577cc467-grs8s Node is not ready 8m34s Warning NodeNotReady pod/dashboard-redirect-76d74cfc9f-9zbjx Node is not ready 8m34s Warning NodeNotReady pod/llmisvc-controller-manager-6df7b6c5d4-fh525 Node is not ready 8m34s Warning NodeNotReady pod/spark-operator-module-controller-manager-59549b5854-ll8qh Node is not ready 8m34s Warning NodeNotReady pod/mlflow-operator-controller-manager-5bbb468f5f-8x672 Node is not ready 8m34s Warning NodeNotReady pod/opendatahub-feast-operator-84ff599545-g2npw Node is not ready 8m34s Warning NodeNotReady pod/kserve-localmodel-controller-manager-fc8c879d7-x4s7n Node is not ready 8m29s Warning NodeNotReady pod/postgres-deployment-86754cb759-5zvbf Node is not ready 8m29s Warning NodeNotReady pod/kserve-localmodelnode-agent-s8gsw Node is not ready 8m29s Warning NodeNotReady pod/dashboard-operator-669d6564bd-52hrd Node is not ready 8m29s Warning NodeNotReady pod/model-registry-operator-controller-manager-758fcb8c6b-79m9p Node is not ready 8m28s Warning NodeNotReady pod/spark-operator-controller-c594b967f-54vsf Node is not ready 8m28s Warning NodeNotReady pod/maas-controller-56c898799-ckvmp Node is not ready 8m28s Warning NodeNotReady pod/dashboard-redirect-76d74cfc9f-m4gjb Node is not ready 8m28s Warning NodeNotReady pod/spark-operator-webhook-5d466b7d66-675wt Node is not ready 8m28s Warning NodeNotReady pod/catalog-controller-manager-76985cd459-sdb9t Node is not ready 8m28s Warning NodeNotReady pod/feast-operator-controller-manager-5fc6bb8d67-ngzc4 Node is not ready 8m28s Warning NodeNotReady pod/rhods-dashboard-78d9976d4f-nqjsf Node is not ready 8m28s Warning NodeNotReady pod/ogx-k8s-operator-controller-manager-59777cdbbb-nbdp2 Node is not ready 8m20s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: skipping Pod rhods-dashboard-78d9976d4f-nqjsf for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-134-105.ec2.internal Not Found 8m12s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: [skipping Pod rhods-dashboard-78d9976d4f-vpd2v for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-141-252.ec2.internal Not Found, skipping Pod rhods-dashboard-78d9976d4f-nqjsf for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-134-105.ec2.internal Not Found] 7m56s Warning FailedToUpdateEndpointSlices service/feast-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-applications/feast-operator-controller-manager-metrics-service: skipping Pod feast-operator-controller-manager-5fc6bb8d67-ngzc4 for Service redhat-ods-applications/feast-operator-controller-manager-metrics-service: Node ip-10-0-134-105.ec2.internal Not Found 7m56s Warning FailedToUpdateEndpointSlices service/dashboard-redirect Error updating Endpoint Slices for Service redhat-ods-applications/dashboard-redirect: skipping Pod dashboard-redirect-76d74cfc9f-m4gjb for Service redhat-ods-applications/dashboard-redirect: Node ip-10-0-134-105.ec2.internal Not Found 7m56s Warning FailedToUpdateEndpointSlices service/ogx-k8s-operator-webhook-service Error updating Endpoint Slices for Service redhat-ods-applications/ogx-k8s-operator-webhook-service: skipping Pod ogx-k8s-operator-controller-manager-59777cdbbb-nbdp2 for Service redhat-ods-applications/ogx-k8s-operator-webhook-service: Node ip-10-0-134-105.ec2.internal Not Found 7m56s Warning FailedToUpdateEndpointSlices service/catalog-webhook-service Error updating Endpoint Slices for Service redhat-ods-applications/catalog-webhook-service: skipping Pod catalog-controller-manager-76985cd459-sdb9t for Service redhat-ods-applications/catalog-webhook-service: Node ip-10-0-134-105.ec2.internal Not Found 7m56s Warning FailedToUpdateEndpointSlices service/maas-controller-webhook-service Error updating Endpoint Slices for Service redhat-ods-applications/maas-controller-webhook-service: skipping Pod maas-controller-56c898799-ckvmp for Service redhat-ods-applications/maas-controller-webhook-service: Node ip-10-0-134-105.ec2.internal Not Found 7m56s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: [skipping Pod rhods-dashboard-78d9976d4f-nqjsf for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-134-105.ec2.internal Not Found, skipping Pod rhods-dashboard-78d9976d4f-vpd2v for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-141-252.ec2.internal Not Found] 7m56s Warning FailedToUpdateEndpointSlices service/maas-controller-metrics Error updating Endpoint Slices for Service redhat-ods-applications/maas-controller-metrics: skipping Pod maas-controller-56c898799-ckvmp for Service redhat-ods-applications/maas-controller-metrics: Node ip-10-0-134-105.ec2.internal Not Found 7m24s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: skipping Pod rhods-dashboard-78d9976d4f-vpd2v for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-141-252.ec2.internal Not Found 2m51s Warning FailedCreate replicaset/automl-ui-74d9b97d49 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m50s Warning FailedCreate replicaset/workbenches-operator-c674d9c9 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m49s Warning FailedCreate replicaset/kuberay-operator-7d87f54798 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m48s Warning FailedCreate replicaset/opendatahub-feast-operator-84ff599545 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m46s Warning FailedCreate replicaset/model-registry-ui-7897899d9b Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m45s Warning FailedCreate replicaset/autorag-ui-58f548cbd Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m44s Warning FailedCreate replicaset/ai-gateway-operator-89f69845b Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m37s Warning FailedCreate replicaset/eval-hub-ui-869b89d9b4 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m36s Warning FailedCreate replicaset/odh-notebook-controller-manager-76dd46fbc Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m36s Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-d6bf47757 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m36s Warning FailedCreate replicaset/spark-operator-module-controller-manager-59549b5854 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m35s Warning FailedCreate replicaset/opendatahub-ogx-operator-644d45d5b Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m28s Warning FailedCreate replicaset/dashboard-redirect-76d74cfc9f Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 51s Warning FailedCreate replicaset/postgres-deployment-86754cb759 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 51s Warning FailedCreate replicaset/mlflow-6f9ddc956d Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- LAST SEEN TYPE REASON OBJECT MESSAGE 2m23s Warning FailedScheduling pod/rhods-operator-7897f9b485-fz5gz no nodes available to schedule pods 2m28s Warning FailedScheduling pod/rhods-operator-7897f9b485-fz5gz no nodes available to schedule pods 7m19s Warning FailedScheduling pod/rhods-operator-7897f9b485-6cfmt 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 119s Warning FailedScheduling pod/rhods-operator-7897f9b485-6cfmt no nodes available to schedule pods 114s Warning FailedScheduling pod/rhods-operator-7897f9b485-6cfmt no nodes available to schedule pods 7m39s Warning FailedScheduling pod/rhods-operator-7897f9b485-fz5gz 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 117m Warning AppliedWithWarnings installplan/install-7tm2n 1 warning(s) generated during installation of operator "rhods-operator.3.6.0-ea.1" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 8m35s Warning NodeNotReady pod/rhods-operator-7897f9b485-mm554 Node is not ready 8m34s Warning NodeNotReady pod/rhods-operator-7897f9b485-d4cbk Node is not ready 8m33s Warning ComponentUnhealthy clusterserviceversion/rhods-operator.3.6.0-ea.1 installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. 8m28s Warning NodeNotReady pod/rhods-operator-7897f9b485-cgn8q Node is not ready 8m20s Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: skipping Pod rhods-operator-7897f9b485-cgn8q for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-134-105.ec2.internal Not Found 8m20s Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: skipping Pod rhods-operator-7897f9b485-cgn8q for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-134-105.ec2.internal Not Found 8m12s Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: [skipping Pod rhods-operator-7897f9b485-d4cbk for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-141-252.ec2.internal Not Found, skipping Pod rhods-operator-7897f9b485-cgn8q for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-134-105.ec2.internal Not Found] 8m12s Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: [skipping Pod rhods-operator-7897f9b485-cgn8q for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-134-105.ec2.internal Not Found, skipping Pod rhods-operator-7897f9b485-d4cbk for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-141-252.ec2.internal Not Found] 8m11s Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: [skipping Pod rhods-operator-7897f9b485-d4cbk for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-141-252.ec2.internal Not Found, skipping Pod rhods-operator-7897f9b485-cgn8q for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-134-105.ec2.internal Not Found] 8m11s Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: [skipping Pod rhods-operator-7897f9b485-cgn8q for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-134-105.ec2.internal Not Found, skipping Pod rhods-operator-7897f9b485-d4cbk for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-141-252.ec2.internal Not Found] 7m39s Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: skipping Pod rhods-operator-7897f9b485-d4cbk for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-141-252.ec2.internal Not Found 7m39s Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: skipping Pod rhods-operator-7897f9b485-d4cbk for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-141-252.ec2.internal Not Found 3m30s Warning InstallCheckFailed clusterserviceversion/rhods-operator.3.6.0-ea.1 install timeout 119s Warning FailedCreate replicaset/rhods-operator-7897f9b485 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found --- namespace/rhoai-model-registries --- LAST SEEN TYPE REASON OBJECT MESSAGE 119s Warning FailedScheduling pod/model-catalog-postgres-7f94f4df48-4kr6j no nodes available to schedule pods 7m19s Warning FailedScheduling pod/model-catalog-postgres-7f94f4df48-4kr6j 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 85m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 86m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 7m19s Warning FailedScheduling pod/model-catalog-dbb95775b-scfvc 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 119s Warning FailedScheduling pod/model-catalog-dbb95775b-scfvc no nodes available to schedule pods 115s Warning FailedScheduling pod/model-catalog-dbb95775b-scfvc no nodes available to schedule pods 115s Warning FailedScheduling pod/model-catalog-postgres-7f94f4df48-4kr6j no nodes available to schedule pods 85m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 86m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 116m Warning FailedMount pod/model-catalog-6fd5d64867-n79qg MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 87m Warning FailedMount pod/model-registry0-5bc586ffb6-n6jsk MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "model-registry0-kube-rbac-proxy" not found 87m Warning FailedMount pod/model-registry0-5bc586ffb6-n6jsk MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-registry0-kube-rbac-proxy-config" not found 87m Warning FailedMount pod/model-registry0-5bc586ffb6-n6jsk MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "model-registry0-kube-rbac-proxy" not found 85m Warning FailedMount pod/model-registry0-65977d544-rpnrq MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "model-registry0-kube-rbac-proxy" not found 85m Warning FailedMount pod/model-registry0-65977d544-rpnrq MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "model-registry0-kube-rbac-proxy" not found 31m Warning Unhealthy pod/model-catalog-6fd5d64867-n79qg Readiness probe failed: Get "http://10.133.0.59:8080/readyz": dial tcp 10.133.0.59:8080: connect: connection refused 31m Warning Unhealthy pod/model-catalog-6fd5d64867-n79qg Readiness probe failed: Get "https://10.133.0.59:8888/healthz": dial tcp 10.133.0.59:8888: connect: connection refused 31m Warning Unhealthy pod/model-catalog-postgres-775cffd8bf-r54fd Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 10m Warning FailedMount pod/model-catalog-dbb95775b-fnwgq MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 10m Warning Unhealthy pod/model-catalog-dbb95775b-fnwgq Readiness probe failed: Get "http://10.132.0.175:8080/readyz": dial tcp 10.132.0.175:8080: connect: connection refused 8m36s Warning NodeNotReady pod/model-catalog-dbb95775b-fnwgq Node is not ready 8m36s Warning NodeNotReady pod/model-catalog-postgres-7f94f4df48-qqp7v Node is not ready --- namespace/rhods-notebooks --- (no events) --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-09-06T15:56:25Z) (no matching error/webhook lines in operator logs for this time window) --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-7897f9b485-6cfmt.log ==================== ==================== OPERATOR POD LOG: rhods-operator-7897f9b485-fz5gz.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/agent-ops-ui-7f9fbf68-msrj6.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/aihub-controller-manager-d877888b8-wsx4s.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/catalog-controller-manager-76985cd459-dnvc8.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-operator-669d6564bd-r5nqx.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-76d74cfc9f-fznfl.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/data-science-pipelines-operator-controller-manager-56b8f89khdkb.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/feast-operator-controller-manager-5fc6bb8d67-nb66k.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/gen-ai-ui-7c7dbc9d78-9dh9f.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-controller-manager-5fbbd6dd88-b729h.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-localmodel-controller-manager-fc8c879d7-fknxk.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-module-controller-manager-66ff7c977-nhf2c.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kubeflow-trainer-controller-manager-5449979c85-cnhbk.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/llmisvc-controller-manager-6df7b6c5d4-qksnx.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-controller-56c898799-x8jbb.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-ui-74675d48f9-cxmkg.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/mlflow-operator-controller-manager-5bbb468f5f-btsd8.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/mlflow-ui-7bb6d46f5f-wd6r5.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-registry-operator-controller-manager-758fcb8c6b-qjrvp.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-serving-api-7df8c9655f-2qdks.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/notebook-controller-deployment-75c8689979-6fn72.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/notebooks-ui-6577cc467-n78h6.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-model-controller-6b9dd8b59d-gqxm5.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/ogx-k8s-operator-controller-manager-59777cdbbb-ncl4z.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/rhods-dashboard-78d9976d4f-f8w8s.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-controller-c594b967f-ffppx.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-webhook-5d466b7d66-jpq9x.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/trainer-operator-controller-manager-6b585cbb9f-dq7fg.log ==================== ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-dbb95775b-scfvc.log ==================== ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-postgres-7f94f4df48-4kr6j.log ====================