==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== OGXReady: status=False reason=Deploying message=Waiting for root OGX operator webhook resources to become ready SparkOperatorReady: status=False reason=DeploymentNotReady message=deployments not ready: spark-operator-controller (desired=1, updated=1, available=0, observedGen=1, gen=1), spark-operator-webhook (desired=1, updated=1, available=0, observedGen=1, gen=1) redhat-ods-applications/agent-ops-ui: ready=0/1 available=0 redhat-ods-applications/ai-gateway-operator: ready=0/1 available=0 redhat-ods-applications/aihub-controller-manager: ready=0/1 available=0 redhat-ods-applications/automl-ui: ready=0/1 available=0 redhat-ods-applications/autorag-ui: ready=0/1 available=0 redhat-ods-applications/catalog-controller-manager: ready=0/1 available=0 redhat-ods-applications/dashboard-operator: ready=0/1 available=0 redhat-ods-applications/dashboard-redirect: ready=0/2 available=0 redhat-ods-applications/data-science-pipelines-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/feast-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/gen-ai-ui: ready=0/1 available=0 redhat-ods-applications/kserve-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-trainer-controller-manager: ready=0/1 available=0 redhat-ods-applications/kuberay-operator: ready=0/1 available=0 redhat-ods-applications/maas-controller: ready=0/1 available=0 redhat-ods-applications/maas-ui: ready=0/1 available=0 redhat-ods-applications/model-registry-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-registry-ui: ready=0/1 available=0 redhat-ods-applications/notebook-controller-deployment: ready=0/1 available=0 redhat-ods-applications/notebooks-ui: ready=0/1 available=0 redhat-ods-applications/odh-notebook-controller-manager: ready=0/1 available=0 redhat-ods-applications/ogx-k8s-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/opendatahub-feast-operator: ready=0/1 available=0 redhat-ods-applications/opendatahub-ogx-operator: ready=0/1 available=0 redhat-ods-applications/rhods-dashboard: ready=0/2 available=0 redhat-ods-applications/spark-operator-controller: ready=0/1 available=0 redhat-ods-applications/spark-operator-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook: ready=0/1 available=0 redhat-ods-applications/trainer-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/workbenches-operator: ready=0/1 available=0 redhat-ods-operator/rhods-operator: ready=0/3 available=0 76s Warning FailedScheduling pod/rhods-dashboard-dd86b5b58-57khm 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 97s Warning FailedScheduling pod/gen-ai-ui-98ccc4f97-pf7h6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/opendatahub-ogx-operator-644d45d5b-bbj2w 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/rhods-dashboard-7764b9895c-8vjj8 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 97s Warning FailedScheduling pod/opendatahub-feast-operator-84ff599545-qjwnx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/agent-ops-ui-687b4b5d4f-znfqj 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/kserve-module-controller-manager-66ff7c977-n9frx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 76s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-ggbks 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/kuberay-operator-7d87f54798-6m9kt 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 75s Warning FailedScheduling pod/workbenches-operator-c674d9c9-bwtk6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-998kd 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 97s Warning FailedScheduling pod/odh-notebook-controller-manager-584b89556c-4bq7j 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7m27s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 95m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-kblb7 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 92m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76s Warning FailedScheduling pod/trainer-operator-controller-manager-6b585cbb9f-9xz9m 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 27m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 95m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m27s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m26s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m26s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m25s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m24s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 4m9s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 4m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m49s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m39s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 95m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75s Warning FailedScheduling pod/automl-ui-74d9b97d49-xwd6p 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76s Warning FailedScheduling pod/autorag-ui-58f548cbd-2gm2x 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98s Warning FailedScheduling pod/notebooks-ui-6577cc467-tfr8f 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 76s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-wrssr 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 76s Warning FailedScheduling pod/rhods-dashboard-54ff89c8f8-v68jq 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 68m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 97s Warning FailedScheduling pod/model-registry-ui-6796658d87-8r98q 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 3m39s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m49s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m26s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m27s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 97s Warning FailedScheduling pod/catalog-controller-manager-6b9685ffd6-vhtrv 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 56m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 95m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98s Warning FailedScheduling pod/dashboard-operator-669d6564bd-d8vj2 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75s Warning FailedScheduling pod/model-registry-operator-controller-manager-7cd9fc56bf-jmdf6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/maas-ui-5bfd6fdb7d-hx8sf 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 3m39s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m49s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 4m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 4m9s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 4m13s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m25s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m45s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76s Warning FailedScheduling pod/dashboard-redirect-5646fb5865-rm57x 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7m21s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m24s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m26s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m26s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 97s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-lbbcs 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7m27s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f896tfvl 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 27m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 114m Warning FailedMount pod/odh-observability-59db79cb46-wlbqf MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 114m Warning FailedMount pod/rhods-dashboard-dd86b5b58-mdhbz MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 114m Warning FailedMount pod/rhods-dashboard-dd86b5b58-rfm99 MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 114m Warning BackOff pod/workbenches-operator-c674d9c9-vf9xm Back-off restarting failed container manager in pod workbenches-operator-c674d9c9-vf9xm_redhat-ods-applications(08707a87-a86c-45b6-a183-5176f0b39f94) 114m Warning FailedMount pod/autorag-ui-58f548cbd-7mjjr MountVolume.SetUp failed for volume "proxy-tls" : secret "autorag-proxy-tls" not found 114m Warning FailedMount pod/gen-ai-ui-98ccc4f97-k8d8g MountVolume.SetUp failed for volume "proxy-tls" : secret "gen-ai-proxy-tls" not found 114m Warning FailedMount pod/maas-ui-5bfd6fdb7d-l5d5l MountVolume.SetUp failed for volume "proxy-tls" : secret "maas-proxy-tls" not found 113m Warning FailedMount pod/odh-model-controller-6b9dd8b59d-9fsj7 MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 113m Warning FailedMount pod/odh-model-controller-6b9dd8b59d-9fsj7 MountVolume.SetUp failed for volume "metrics-certs" : secret "odh-model-controller-metrics-tls" not found 113m Warning Unhealthy pod/rhods-dashboard-dd86b5b58-mdhbz Readiness probe failed: Get "https://10.134.0.58:8943/healthcheck": dial tcp 10.134.0.58:8943: connect: connection refused 113m Warning Unhealthy pod/rhods-dashboard-dd86b5b58-mdhbz Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 112m Warning Unhealthy pod/rhods-dashboard-dd86b5b58-mdhbz Readiness probe failed: Get "https://10.134.0.58:8444/healthz": dial tcp 10.134.0.58:8444: connect: connection refused 112m Warning Unhealthy pod/rhods-dashboard-dd86b5b58-rfm99 Readiness probe failed: Get "https://10.132.0.44:8943/healthcheck": dial tcp 10.132.0.44:8943: connect: connection refused 112m Warning Unhealthy pod/rhods-dashboard-dd86b5b58-rfm99 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 112m Warning Unhealthy pod/kubeflow-trainer-controller-manager-5449979c85-w7bxp Readiness probe failed: HTTP probe failed with statuscode: 500 112m Warning Unhealthy pod/rhods-dashboard-dd86b5b58-rfm99 Readiness probe failed: Get "https://10.132.0.44:8444/healthz": dial tcp 10.132.0.44:8444: connect: connection refused 95m Warning FailedMount pod/ogx-k8s-operator-controller-manager-59777cdbbb-5ccmx MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 7m25s Warning Unhealthy pod/rhods-dashboard-54ff89c8f8-4vc5b Readiness probe failed: Get "https://10.133.0.60:8943/healthcheck": dial tcp 10.133.0.60:8943: connect: connection refused 7m24s Warning Unhealthy pod/rhods-dashboard-54ff89c8f8-4vc5b Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 7m15s Warning Unhealthy pod/rhods-dashboard-54ff89c8f8-z5rhf Readiness probe failed: Get "https://10.132.0.48:8943/healthcheck": dial tcp 10.132.0.48:8943: connect: connection refused 7m15s Warning Unhealthy pod/rhods-dashboard-54ff89c8f8-z5rhf Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 7m Warning Unhealthy pod/rhods-dashboard-54ff89c8f8-4vc5b Readiness probe failed: Get "https://10.133.0.60:8444/healthz": dial tcp 10.133.0.60:8444: connect: connection refused 6m55s Warning Unhealthy pod/rhods-dashboard-54ff89c8f8-z5rhf Readiness probe failed: Get "https://10.132.0.48:8444/healthz": dial tcp 10.132.0.48:8444: connect: connection refused 4m31s Warning FailedMount pod/agent-ops-ui-687b4b5d4f-dzp72 MountVolume.SetUp failed for volume "proxy-tls" : secret "agent-ops-proxy-tls" not found 3m57s Warning FailedMount pod/maas-controller-56c898799-d5rjs MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found 3m49s Warning FailedMount pod/model-registry-operator-controller-manager-7cd9fc56bf-4skr5 MountVolume.SetUp failed for volume "server-cert" : secret "model-registry-operator-controller-manager-metrics-service" not found 3m49s Warning FailedMount pod/model-registry-operator-controller-manager-7cd9fc56bf-4skr5 MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 3m39s Warning Unhealthy pod/rhods-dashboard-dd86b5b58-zcw84 Readiness probe failed: Get "https://10.133.0.79:8943/healthcheck": dial tcp 10.133.0.79:8943: connect: connection refused 3m39s Warning Unhealthy pod/rhods-dashboard-dd86b5b58-zcw84 Readiness probe failed: Get "https://10.133.0.79:8444/healthz": dial tcp 10.133.0.79:8444: connect: connection refused 3m39s Warning Unhealthy pod/rhods-dashboard-dd86b5b58-zcw84 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2m38s Warning NodeNotReady pod/notebooks-ui-6577cc467-8vft9 Node is not ready 2m38s Warning NodeNotReady pod/dashboard-redirect-5646fb5865-vsp6b Node is not ready 2m38s Warning NodeNotReady pod/autorag-ui-58f548cbd-r9n9c Node is not ready 2m38s Warning NodeNotReady pod/opendatahub-ogx-operator-644d45d5b-s44js Node is not ready 2m38s Warning NodeNotReady pod/rhods-dashboard-7764b9895c-7nm7h Node is not ready 2m38s Warning NodeNotReady pod/agent-ops-ui-687b4b5d4f-dzp72 Node is not ready 2m38s Warning NodeNotReady pod/dashboard-operator-669d6564bd-8844g Node is not ready 2m38s Warning NodeNotReady pod/trainer-operator-controller-manager-6b585cbb9f-twg7f Node is not ready 2m38s Warning NodeNotReady pod/dashboard-redirect-5646fb5865-tjf26 Node is not ready 2m38s Warning NodeNotReady pod/maas-controller-56c898799-d5rjs Node is not ready 2m38s Warning NodeNotReady pod/spark-operator-module-controller-manager-59549b5854-7t9sn Node is not ready 2m38s Warning NodeNotReady pod/kubeflow-trainer-controller-manager-5449979c85-mf8wx Node is not ready 2m38s Warning NodeNotReady pod/automl-ui-74d9b97d49-kqvs5 Node is not ready 2m38s Warning NodeNotReady pod/kuberay-operator-7d87f54798-4zw86 Node is not ready 2m38s Warning NodeNotReady pod/opendatahub-feast-operator-84ff599545-gvx6q Node is not ready 2m38s Warning NodeNotReady pod/rhods-dashboard-dd86b5b58-9thsf Node is not ready 2m38s Warning NodeNotReady pod/odh-notebook-controller-manager-584b89556c-gp52l Node is not ready 2m38s Warning NodeNotReady pod/data-science-pipelines-operator-controller-manager-56b8f89hf6hz Node is not ready 2m38s Warning NodeNotReady pod/maas-ui-5bfd6fdb7d-ptttk Node is not ready 2m38s Warning NodeNotReady pod/feast-operator-controller-manager-5fc6bb8d67-h5gkp Node is not ready 2m38s Warning NodeNotReady pod/gen-ai-ui-98ccc4f97-475x8 Node is not ready 2m38s Warning NodeNotReady pod/ai-gateway-operator-89f69845b-6t557 Node is not ready 2m38s Warning NodeNotReady pod/notebook-controller-deployment-75c8689979-j5sbz Node is not ready 2m38s Warning NodeNotReady pod/model-registry-operator-controller-manager-7cd9fc56bf-4skr5 Node is not ready 2m38s Warning NodeNotReady pod/model-registry-ui-6796658d87-ttmd5 Node is not ready 2m37s Warning NodeNotReady pod/aihub-controller-manager-d877888b8-bgrx6 Node is not ready 2m37s Warning NodeNotReady pod/kserve-module-controller-manager-66ff7c977-hd6c7 Node is not ready 2m37s Warning NodeNotReady pod/workbenches-operator-c674d9c9-c2hgq Node is not ready 98s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: [skipping Pod rhods-dashboard-54ff89c8f8-sb25f for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-143-50.ec2.internal Not Found, skipping Pod rhods-dashboard-dd86b5b58-9thsf for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-143-50.ec2.internal Not Found] 90s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: [skipping Pod rhods-dashboard-dd86b5b58-9thsf for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-143-50.ec2.internal Not Found, skipping Pod rhods-dashboard-54ff89c8f8-sb25f for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-143-50.ec2.internal Not Found] 77s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: skipping Pod rhods-dashboard-54ff89c8f8-sb25f for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-143-50.ec2.internal Not Found 97s Warning FailedScheduling pod/rhods-operator-7fdc98ddcb-l8b4q 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 77s Warning FailedScheduling pod/rhods-operator-7fdc98ddcb-h9n9p 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115m Warning AppliedWithWarnings installplan/install-9kwvf 1 warning(s) generated during installation of operator "rhods-operator.3.6.0-ea.1" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 2m38s Warning NodeNotReady pod/rhods-operator-7fdc98ddcb-lzxsq Node is not ready 2m38s Warning NodeNotReady pod/rhods-operator-7fdc98ddcb-fr9gc Node is not ready 2m38s Warning NodeNotReady pod/rhods-operator-7fdc98ddcb-5rq8w Node is not ready 2m37s Warning ComponentUnhealthy clusterserviceversion/rhods-operator.3.6.0-ea.1 installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. 89s Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: skipping Pod rhods-operator-7fdc98ddcb-lzxsq for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-143-50.ec2.internal Not Found 89s Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: skipping Pod rhods-operator-7fdc98ddcb-lzxsq for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-143-50.ec2.internal Not Found 25m Warning Deleting modelregistry/e2e-test-registry-298031 Custom Resource e2e-test-registry-298031 is being deleted from the namespace rhoai-model-registries 21m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 24m Warning Deleting modelregistry/postgres-registry-298031 Custom Resource postgres-registry-298031 is being deleted from the namespace rhoai-model-registries 23m Warning Deleting modelregistry/default-db-registry-298031 Custom Resource default-db-registry-298031 is being deleted from the namespace rhoai-model-registries 21m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 25m Warning Deleting modelregistry/e2e-test-registry-298031 Custom Resource e2e-test-registry-298031 is being deleted from the namespace rhoai-model-registries 23m Warning Deleting modelregistry/default-db-registry-298031 Custom Resource default-db-registry-298031 is being deleted from the namespace rhoai-model-registries 113m Warning FailedMount pod/model-catalog-b5b55bdfb-m889w MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 26m Warning FailedMount pod/e2e-test-registry-298031-65b876d98d-q75j8 MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "e2e-test-registry-298031-kube-rbac-proxy" not found 26m Warning FailedMount pod/e2e-test-registry-298031-65b876d98d-q75j8 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "e2e-test-registry-298031-kube-rbac-proxy-config" not found 26m Warning FailedMount pod/e2e-test-registry-298031-65b876d98d-q75j8 MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "e2e-test-registry-298031-kube-rbac-proxy" not found 24m Warning FailedMount pod/default-db-registry-298031-6fdccf6498-qc68v MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "default-db-registry-298031-kube-rbac-proxy" not found 24m Warning FailedMount pod/default-db-registry-298031-6fdccf6498-qc68v MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "default-db-registry-298031-kube-rbac-proxy" not found 24m Warning Unhealthy pod/default-db-registry-298031-6fdccf6498-qc68v Startup probe failed: Get "http://10.132.0.94:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 24m Warning Unhealthy pod/default-db-registry-298031-6fdccf6498-qc68v Startup probe failed: HTTP probe failed with statuscode: 503 23m Warning FailedMount pod/test-model-registry-662874-55c7499d55-x5wb6 MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "test-model-registry-662874-kube-rbac-proxy" not found 23m Warning FailedMount pod/test-model-registry-662874-55c7499d55-x5wb6 MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "test-model-registry-662874-kube-rbac-proxy" not found 21m Warning Unhealthy pod/model-registry-db-662874-f958cd66c-wthfh Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 6m59s Warning Unhealthy pod/model-catalog-postgres-75fcbb584-bchkw Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 6m57s Warning Unhealthy pod/model-catalog-b5b55bdfb-m889w Readiness probe failed: Get "http://10.134.0.60:8080/readyz": dial tcp 10.134.0.60:8080: connect: connection refused 6m57s Warning Unhealthy pod/model-catalog-b5b55bdfb-m889w Readiness probe failed: Get "https://10.134.0.60:8888/healthz": dial tcp 10.134.0.60:8888: connect: connection refused --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-09-09T21:46:25Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.21.31 Kubernetes Version: v1.34.9 --- Namespaces --- namespace/opendatahub namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.3 Authorino Operator 1.4.3 authorino-operator.v1.4.2 Installing cert-manager-operator.v1.20.0 cert-manager Operator for Red Hat OpenShift 1.20.0 cert-manager-operator.v1.19.0 Installing custom-metrics-autoscaler.v2.19.0-3 Custom Metrics Autoscaler 2.19.0-3 custom-metrics-autoscaler.v2.19.0-2 Installing dns-operator.v1.4.2 DNS Operator 1.4.2 dns-operator.v1.4.1 Installing kueue-operator.v1.4.2 Red Hat build of Kueue 1.4.2 kueue-operator.v1.4.1 Installing limitador-operator.v1.4.2 Limitador Operator 1.4.2 limitador-operator.v1.4.1 Installing rhcl-operator.v1.4.3 Red Hat Connectivity Link 1.4.3 rhcl-operator.v1.4.2 Installing rhods-operator.3.6.0-ea.1 Red Hat OpenShift AI 3.6.0-ea.1 Installing --- CSVs not in Succeeded phase --- authorino-operator.v1.4.3: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace cert-manager-operator.v1.20.0: phase=Installing reason=Copied message=The operator is running in cert-manager-operator but is managing this namespace custom-metrics-autoscaler.v2.19.0-3: phase=Installing reason=Copied message=The operator is running in openshift-keda but is managing this namespace dns-operator.v1.4.2: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace kueue-operator.v1.4.2: phase=Installing reason=Copied message=The operator is running in openshift-kueue-operator but is managing this namespace limitador-operator.v1.4.2: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace rhcl-operator.v1.4.3: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace rhods-operator.3.6.0-ea.1: phase=Installing reason=InstallWaiting message=installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc False NotReady DSC Ready summary: False reason=NotReady DSC component states: aigateway: managementState=Managed aipipelines: managementState=Managed batchGateway: managementState=Removed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Removed llamastackoperator: managementState=n/a mcplifecycleoperator: managementState=Removed mlflowoperator: managementState=Removed modelregistry: managementState=Managed modelsAsAService: managementState=Managed ogx: managementState=Managed ray: managementState=Managed sparkoperator: managementState=Managed trainer: managementState=Managed trainingoperator: managementState=Removed trustyai: managementState=Removed workbenches: managementState=Managed workbenchesV2: managementState=Removed DSC conditions: Ready: status=False reason=NotReady message=Some modules are not ready: kserve, modelregistry, ogx, sparkoperator; degraded: dashboard ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= ModulesReady: status=False reason=NotReady message=Some modules are not ready: kserve, modelregistry, ogx, sparkoperator; degraded: dashboard AIGatewayReady: status=True reason= message= AIPipelinesReady: status=True reason= message= BatchGatewayReady: status=False reason=Removed message=Submodule ManagementState is set to Removed DashboardReady: status=True reason=AllDependentsHealthy message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=False reason=AwaitingReadiness message=Submodule is enabled (Managed) but the module operator has not reported its status yet KserveLLMInferenceServiceWideEPDependencies: status=False reason=AwaitingReadiness message=Submodule is enabled (Managed) but the module operator has not reported its status yet KserveReady: status=False reason=NotReady message=Module has not reported a Ready condition yet KueueReady: status=False reason=Removed message=Component ManagementState is set to Removed MCPLifecycleOperatorReady: status=False reason=Removed message=Module ManagementState is set to Removed MLflowOperatorReady: status=False reason=Removed message=Module ManagementState is set to Removed ModelRegistryReady: status=False reason=ChildDeploymentNotReady message=child deployment catalog-controller-manager not yet Available ModelsAsAServiceReady: status=True reason=Ready message=modelsAsAService is Managed and deployments are available OGXReady: status=False reason=Deploying message=Waiting for root OGX operator webhook resources to become ready RayReady: status=True reason= message= SparkOperatorReady: status=False reason=DeploymentNotReady message=deployments not ready: spark-operator-controller (desired=1, updated=1, available=0, observedGen=1, gen=1), spark-operator-webhook (desired=1, updated=1, available=0, observedGen=1, gen=1) TrainerReady: status=True reason= message= TrustyAIReady: status=False reason=Removed message=Component ManagementState is set to Removed WorkbenchesReady: status=True reason=ReconcileSuccess message=Workbenches component is ready WorkbenchesV2Ready: status=False reason=Removed message=Submodule ManagementState is set to Removed --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 114m Ready 2026-09-09T19:51:37Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=False reason=Removed message=Monitoring is not enabled MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in Monitoring CR OpenTelemetryCollectorAvailable: status=False reason=MetricsAndTracesNotConfigured message=Metrics and traces are not configured in Monitoring CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in Monitoring CR NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in Monitoring CR PersesAvailable: status=False reason=MetricsAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured ProvisioningSucceeded: status=True reason=Available message= Ready: status=True reason=Available message= --- Component CRs (Ready) --- NAME READY REASON VERSION aigateway.components.platform.opendatahub.io/default-aigateway True 1.26.2 NAME APPNAMESPACE AGE aihub.components.platform.opendatahub.io/default-aihub redhat-ods-applications 4m17s NAME PHASE READY REASON URL AGE dashboard.components.platform.opendatahub.io/default-dashboard Ready True AllDependentsHealthy https://rh-ai.apps.98f58480886a49c10614.prod.konflux-ocp-ci.dev/ 4m58s NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve NAME READY REASON DISTRIBUTION ogx.components.platform.opendatahub.io/default-ogx False Deploying Red Hat OpenShift AI NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON sparkoperator.components.platform.opendatahub.io/default-sparkoperator False DeploymentNotReady NAME READY REASON AGE trainer.components.platform.opendatahub.io/default-trainer True 4m17s NAME READY REASON PHASE workbenches.components.platform.opendatahub.io/default-workbenches True ReconcileSuccess Ready --- Deployments not fully ready (ODH/RHOAI namespaces) --- redhat-ods-applications/agent-ops-ui: ready=0/1 available=0 redhat-ods-applications/ai-gateway-operator: ready=0/1 available=0 redhat-ods-applications/aihub-controller-manager: ready=0/1 available=0 redhat-ods-applications/automl-ui: ready=0/1 available=0 redhat-ods-applications/autorag-ui: ready=0/1 available=0 redhat-ods-applications/catalog-controller-manager: ready=0/1 available=0 redhat-ods-applications/dashboard-operator: ready=0/1 available=0 redhat-ods-applications/dashboard-redirect: ready=0/2 available=0 redhat-ods-applications/data-science-pipelines-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/feast-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/gen-ai-ui: ready=0/1 available=0 redhat-ods-applications/kserve-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-trainer-controller-manager: ready=0/1 available=0 redhat-ods-applications/kuberay-operator: ready=0/1 available=0 redhat-ods-applications/maas-controller: ready=0/1 available=0 redhat-ods-applications/maas-ui: ready=0/1 available=0 redhat-ods-applications/model-registry-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-registry-ui: ready=0/1 available=0 redhat-ods-applications/notebook-controller-deployment: ready=0/1 available=0 redhat-ods-applications/notebooks-ui: ready=0/1 available=0 redhat-ods-applications/odh-notebook-controller-manager: ready=0/1 available=0 redhat-ods-applications/ogx-k8s-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/opendatahub-feast-operator: ready=0/1 available=0 redhat-ods-applications/opendatahub-ogx-operator: ready=0/1 available=0 redhat-ods-applications/rhods-dashboard: ready=0/2 available=0 redhat-ods-applications/spark-operator-controller: ready=0/1 available=0 redhat-ods-applications/spark-operator-module-controller-manager: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook: ready=0/1 available=0 redhat-ods-applications/trainer-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/workbenches-operator: ready=0/1 available=0 redhat-ods-operator/rhods-operator: ready=0/3 available=0 --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- redhat-ods-applications/agent-ops-ui-687b4b5d4f-znfqj: phase=Pending reason= redhat-ods-applications/ai-gateway-operator-89f69845b-998kd: phase=Pending reason= redhat-ods-applications/aihub-controller-manager-d877888b8-kblb7: phase=Pending reason= redhat-ods-applications/automl-ui-74d9b97d49-xwd6p: phase=Pending reason= redhat-ods-applications/autorag-ui-58f548cbd-2gm2x: phase=Pending reason= redhat-ods-applications/catalog-controller-manager-6b9685ffd6-vhtrv: phase=Pending reason= redhat-ods-applications/dashboard-operator-669d6564bd-d8vj2: phase=Pending reason= redhat-ods-applications/dashboard-redirect-5646fb5865-rm57x: phase=Pending reason= redhat-ods-applications/data-science-pipelines-operator-controller-manager-56b8f896tfvl: phase=Pending reason= redhat-ods-applications/gen-ai-ui-98ccc4f97-pf7h6: phase=Pending reason= redhat-ods-applications/kserve-module-controller-manager-66ff7c977-n9frx: phase=Pending reason= redhat-ods-applications/kubeflow-trainer-controller-manager-5449979c85-ggbks: phase=Pending reason= redhat-ods-applications/kuberay-operator-7d87f54798-6m9kt: phase=Pending reason= redhat-ods-applications/maas-ui-5bfd6fdb7d-hx8sf: phase=Pending reason= redhat-ods-applications/model-registry-operator-controller-manager-7cd9fc56bf-jmdf6: phase=Pending reason= redhat-ods-applications/model-registry-ui-6796658d87-8r98q: phase=Pending reason= redhat-ods-applications/notebook-controller-deployment-75c8689979-wrssr: phase=Pending reason= redhat-ods-applications/notebooks-ui-6577cc467-tfr8f: phase=Pending reason= redhat-ods-applications/odh-notebook-controller-manager-584b89556c-4bq7j: phase=Pending reason= redhat-ods-applications/opendatahub-feast-operator-84ff599545-qjwnx: phase=Pending reason= redhat-ods-applications/opendatahub-ogx-operator-644d45d5b-bbj2w: phase=Pending reason= redhat-ods-applications/rhods-dashboard-54ff89c8f8-v68jq: phase=Pending reason= redhat-ods-applications/rhods-dashboard-7764b9895c-8vjj8: phase=Pending reason= redhat-ods-applications/rhods-dashboard-dd86b5b58-57khm: phase=Pending reason= redhat-ods-applications/spark-operator-module-controller-manager-59549b5854-lbbcs: phase=Pending reason= redhat-ods-applications/trainer-operator-controller-manager-6b585cbb9f-9xz9m: phase=Pending reason= redhat-ods-applications/workbenches-operator-c674d9c9-bwtk6: phase=Pending reason= redhat-ods-operator/rhods-operator-7fdc98ddcb-h9n9p: phase=Pending reason= redhat-ods-operator/rhods-operator-7fdc98ddcb-l8b4q: phase=Pending reason= --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/opendatahub --- (no events) --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 76s Warning FailedScheduling pod/rhods-dashboard-dd86b5b58-57khm 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 97s Warning FailedScheduling pod/gen-ai-ui-98ccc4f97-pf7h6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/opendatahub-ogx-operator-644d45d5b-bbj2w 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/rhods-dashboard-7764b9895c-8vjj8 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 97s Warning FailedScheduling pod/opendatahub-feast-operator-84ff599545-qjwnx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/agent-ops-ui-687b4b5d4f-znfqj 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/kserve-module-controller-manager-66ff7c977-n9frx 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 76s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-5449979c85-ggbks 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/kuberay-operator-7d87f54798-6m9kt 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 75s Warning FailedScheduling pod/workbenches-operator-c674d9c9-bwtk6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/ai-gateway-operator-89f69845b-998kd 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 97s Warning FailedScheduling pod/odh-notebook-controller-manager-584b89556c-4bq7j 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7m27s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 95m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76s Warning FailedScheduling pod/aihub-controller-manager-d877888b8-kblb7 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 92m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76s Warning FailedScheduling pod/trainer-operator-controller-manager-6b585cbb9f-9xz9m 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 27m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 95m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m27s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m26s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m26s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m25s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m24s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 4m9s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 4m Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m49s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m39s Warning InvalidConfiguration servicemonitor/aihub-controller-manager-metrics-monitor "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 95m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75s Warning FailedScheduling pod/automl-ui-74d9b97d49-xwd6p 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76s Warning FailedScheduling pod/autorag-ui-58f548cbd-2gm2x 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 99m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98s Warning FailedScheduling pod/notebooks-ui-6577cc467-tfr8f 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 76s Warning FailedScheduling pod/notebook-controller-deployment-75c8689979-wrssr 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 76s Warning FailedScheduling pod/rhods-dashboard-54ff89c8f8-v68jq 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 68m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 97s Warning FailedScheduling pod/model-registry-ui-6796658d87-8r98q 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 3m39s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m49s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m26s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m27s Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 97s Warning FailedScheduling pod/catalog-controller-manager-6b9685ffd6-vhtrv 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 56m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 68m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 95m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 98s Warning FailedScheduling pod/dashboard-operator-669d6564bd-d8vj2 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 99m Warning InvalidConfiguration servicemonitor/model-registry-operator-controller-manager-metrics-monitor "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 75s Warning FailedScheduling pod/model-registry-operator-controller-manager-7cd9fc56bf-jmdf6 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 98s Warning FailedScheduling pod/maas-ui-5bfd6fdb7d-hx8sf 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 3m39s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 3m49s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 4m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 4m9s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 4m13s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m25s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 6m45s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76s Warning FailedScheduling pod/dashboard-redirect-5646fb5865-rm57x 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7m21s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m24s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m26s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 7m26s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 97s Warning FailedScheduling pod/spark-operator-module-controller-manager-59549b5854-lbbcs 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 7m27s Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 13m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 26m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 27m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 76s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-56b8f896tfvl 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 27m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 56m Warning InvalidConfiguration servicemonitor/maas-controller-metrics "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 114m Warning BadConfig certificaterequest/odh-observability-webhook-cert-1 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients 114m Warning FailedMount pod/odh-observability-59db79cb46-wlbqf MountVolume.SetUp failed for volume "webhook-certs" : secret "odh-observability-webhook-cert" not found 114m Warning FailedMount pod/rhods-dashboard-dd86b5b58-mdhbz MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 114m Warning FailedMount pod/rhods-dashboard-dd86b5b58-rfm99 MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 114m Warning BackOff pod/workbenches-operator-c674d9c9-vf9xm Back-off restarting failed container manager in pod workbenches-operator-c674d9c9-vf9xm_redhat-ods-applications(08707a87-a86c-45b6-a183-5176f0b39f94) 114m Warning FailedMount pod/autorag-ui-58f548cbd-7mjjr MountVolume.SetUp failed for volume "proxy-tls" : secret "autorag-proxy-tls" not found 114m Warning FailedMount pod/gen-ai-ui-98ccc4f97-k8d8g MountVolume.SetUp failed for volume "proxy-tls" : secret "gen-ai-proxy-tls" not found 114m Warning FailedMount pod/maas-ui-5bfd6fdb7d-l5d5l MountVolume.SetUp failed for volume "proxy-tls" : secret "maas-proxy-tls" not found 113m Warning FailedMount pod/odh-model-controller-6b9dd8b59d-9fsj7 MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 113m Warning FailedMount pod/odh-model-controller-6b9dd8b59d-9fsj7 MountVolume.SetUp failed for volume "metrics-certs" : secret "odh-model-controller-metrics-tls" not found 113m Warning Unhealthy pod/rhods-dashboard-dd86b5b58-mdhbz Readiness probe failed: Get "https://10.134.0.58:8943/healthcheck": dial tcp 10.134.0.58:8943: connect: connection refused 113m Warning Unhealthy pod/rhods-dashboard-dd86b5b58-mdhbz Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 112m Warning Unhealthy pod/rhods-dashboard-dd86b5b58-mdhbz Readiness probe failed: Get "https://10.134.0.58:8444/healthz": dial tcp 10.134.0.58:8444: connect: connection refused 112m Warning Unhealthy pod/rhods-dashboard-dd86b5b58-rfm99 Readiness probe failed: Get "https://10.132.0.44:8943/healthcheck": dial tcp 10.132.0.44:8943: connect: connection refused 112m Warning Unhealthy pod/rhods-dashboard-dd86b5b58-rfm99 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 112m Warning Unhealthy pod/kubeflow-trainer-controller-manager-5449979c85-w7bxp Readiness probe failed: HTTP probe failed with statuscode: 500 112m Warning Unhealthy pod/rhods-dashboard-dd86b5b58-rfm99 Readiness probe failed: Get "https://10.132.0.44:8444/healthz": dial tcp 10.132.0.44:8444: connect: connection refused 95m Warning FailedMount pod/ogx-k8s-operator-controller-manager-59777cdbbb-5ccmx MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 7m25s Warning Unhealthy pod/rhods-dashboard-54ff89c8f8-4vc5b Readiness probe failed: Get "https://10.133.0.60:8943/healthcheck": dial tcp 10.133.0.60:8943: connect: connection refused 7m24s Warning Unhealthy pod/rhods-dashboard-54ff89c8f8-4vc5b Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 7m15s Warning Unhealthy pod/rhods-dashboard-54ff89c8f8-z5rhf Readiness probe failed: Get "https://10.132.0.48:8943/healthcheck": dial tcp 10.132.0.48:8943: connect: connection refused 7m15s Warning Unhealthy pod/rhods-dashboard-54ff89c8f8-z5rhf Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 7m Warning Unhealthy pod/rhods-dashboard-54ff89c8f8-4vc5b Readiness probe failed: Get "https://10.133.0.60:8444/healthz": dial tcp 10.133.0.60:8444: connect: connection refused 6m55s Warning Unhealthy pod/rhods-dashboard-54ff89c8f8-z5rhf Readiness probe failed: Get "https://10.132.0.48:8444/healthz": dial tcp 10.132.0.48:8444: connect: connection refused 4m31s Warning FailedMount pod/agent-ops-ui-687b4b5d4f-dzp72 MountVolume.SetUp failed for volume "proxy-tls" : secret "agent-ops-proxy-tls" not found 3m57s Warning FailedMount pod/maas-controller-56c898799-d5rjs MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found 3m49s Warning FailedMount pod/model-registry-operator-controller-manager-7cd9fc56bf-4skr5 MountVolume.SetUp failed for volume "server-cert" : secret "model-registry-operator-controller-manager-metrics-service" not found 3m49s Warning FailedMount pod/model-registry-operator-controller-manager-7cd9fc56bf-4skr5 MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 3m39s Warning Unhealthy pod/rhods-dashboard-dd86b5b58-zcw84 Readiness probe failed: Get "https://10.133.0.79:8943/healthcheck": dial tcp 10.133.0.79:8943: connect: connection refused 3m39s Warning Unhealthy pod/rhods-dashboard-dd86b5b58-zcw84 Readiness probe failed: Get "https://10.133.0.79:8444/healthz": dial tcp 10.133.0.79:8444: connect: connection refused 3m39s Warning Unhealthy pod/rhods-dashboard-dd86b5b58-zcw84 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2m38s Warning NodeNotReady pod/notebooks-ui-6577cc467-8vft9 Node is not ready 2m38s Warning NodeNotReady pod/dashboard-redirect-5646fb5865-vsp6b Node is not ready 2m38s Warning NodeNotReady pod/autorag-ui-58f548cbd-r9n9c Node is not ready 2m38s Warning NodeNotReady pod/opendatahub-ogx-operator-644d45d5b-s44js Node is not ready 2m38s Warning NodeNotReady pod/rhods-dashboard-7764b9895c-7nm7h Node is not ready 2m38s Warning NodeNotReady pod/agent-ops-ui-687b4b5d4f-dzp72 Node is not ready 2m38s Warning NodeNotReady pod/dashboard-operator-669d6564bd-8844g Node is not ready 2m38s Warning NodeNotReady pod/trainer-operator-controller-manager-6b585cbb9f-twg7f Node is not ready 2m38s Warning NodeNotReady pod/dashboard-redirect-5646fb5865-tjf26 Node is not ready 2m38s Warning NodeNotReady pod/maas-controller-56c898799-d5rjs Node is not ready 2m38s Warning NodeNotReady pod/spark-operator-module-controller-manager-59549b5854-7t9sn Node is not ready 2m38s Warning NodeNotReady pod/kubeflow-trainer-controller-manager-5449979c85-mf8wx Node is not ready 2m38s Warning NodeNotReady pod/automl-ui-74d9b97d49-kqvs5 Node is not ready 2m38s Warning NodeNotReady pod/kuberay-operator-7d87f54798-4zw86 Node is not ready 2m38s Warning NodeNotReady pod/opendatahub-feast-operator-84ff599545-gvx6q Node is not ready 2m38s Warning NodeNotReady pod/rhods-dashboard-dd86b5b58-9thsf Node is not ready 2m38s Warning NodeNotReady pod/odh-notebook-controller-manager-584b89556c-gp52l Node is not ready 2m38s Warning NodeNotReady pod/data-science-pipelines-operator-controller-manager-56b8f89hf6hz Node is not ready 2m38s Warning NodeNotReady pod/maas-ui-5bfd6fdb7d-ptttk Node is not ready 2m38s Warning NodeNotReady pod/feast-operator-controller-manager-5fc6bb8d67-h5gkp Node is not ready 2m38s Warning NodeNotReady pod/gen-ai-ui-98ccc4f97-475x8 Node is not ready 2m38s Warning NodeNotReady pod/ai-gateway-operator-89f69845b-6t557 Node is not ready 2m38s Warning NodeNotReady pod/notebook-controller-deployment-75c8689979-j5sbz Node is not ready 2m38s Warning NodeNotReady pod/model-registry-operator-controller-manager-7cd9fc56bf-4skr5 Node is not ready 2m38s Warning NodeNotReady pod/model-registry-ui-6796658d87-ttmd5 Node is not ready 2m37s Warning NodeNotReady pod/aihub-controller-manager-d877888b8-bgrx6 Node is not ready 2m37s Warning NodeNotReady pod/kserve-module-controller-manager-66ff7c977-hd6c7 Node is not ready 2m37s Warning NodeNotReady pod/workbenches-operator-c674d9c9-c2hgq Node is not ready 98s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: [skipping Pod rhods-dashboard-54ff89c8f8-sb25f for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-143-50.ec2.internal Not Found, skipping Pod rhods-dashboard-dd86b5b58-9thsf for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-143-50.ec2.internal Not Found] 90s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: [skipping Pod rhods-dashboard-dd86b5b58-9thsf for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-143-50.ec2.internal Not Found, skipping Pod rhods-dashboard-54ff89c8f8-sb25f for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-143-50.ec2.internal Not Found] 77s Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: skipping Pod rhods-dashboard-54ff89c8f8-sb25f for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-143-50.ec2.internal Not Found --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- LAST SEEN TYPE REASON OBJECT MESSAGE 97s Warning FailedScheduling pod/rhods-operator-7fdc98ddcb-l8b4q 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 77s Warning FailedScheduling pod/rhods-operator-7fdc98ddcb-h9n9p 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 115m Warning AppliedWithWarnings installplan/install-9kwvf 1 warning(s) generated during installation of operator "rhods-operator.3.6.0-ea.1" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 2m38s Warning NodeNotReady pod/rhods-operator-7fdc98ddcb-lzxsq Node is not ready 2m38s Warning NodeNotReady pod/rhods-operator-7fdc98ddcb-fr9gc Node is not ready 2m38s Warning NodeNotReady pod/rhods-operator-7fdc98ddcb-5rq8w Node is not ready 2m37s Warning ComponentUnhealthy clusterserviceversion/rhods-operator.3.6.0-ea.1 installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. 89s Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: skipping Pod rhods-operator-7fdc98ddcb-lzxsq for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-143-50.ec2.internal Not Found 89s Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: skipping Pod rhods-operator-7fdc98ddcb-lzxsq for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-143-50.ec2.internal Not Found --- namespace/rhoai-model-registries --- LAST SEEN TYPE REASON OBJECT MESSAGE 25m Warning Deleting modelregistry/e2e-test-registry-298031 Custom Resource e2e-test-registry-298031 is being deleted from the namespace rhoai-model-registries 21m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 24m Warning Deleting modelregistry/postgres-registry-298031 Custom Resource postgres-registry-298031 is being deleted from the namespace rhoai-model-registries 23m Warning Deleting modelregistry/default-db-registry-298031 Custom Resource default-db-registry-298031 is being deleted from the namespace rhoai-model-registries 21m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 25m Warning Deleting modelregistry/e2e-test-registry-298031 Custom Resource e2e-test-registry-298031 is being deleted from the namespace rhoai-model-registries 23m Warning Deleting modelregistry/default-db-registry-298031 Custom Resource default-db-registry-298031 is being deleted from the namespace rhoai-model-registries 113m Warning FailedMount pod/model-catalog-b5b55bdfb-m889w MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 26m Warning FailedMount pod/e2e-test-registry-298031-65b876d98d-q75j8 MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "e2e-test-registry-298031-kube-rbac-proxy" not found 26m Warning FailedMount pod/e2e-test-registry-298031-65b876d98d-q75j8 MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "e2e-test-registry-298031-kube-rbac-proxy-config" not found 26m Warning FailedMount pod/e2e-test-registry-298031-65b876d98d-q75j8 MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "e2e-test-registry-298031-kube-rbac-proxy" not found 24m Warning FailedMount pod/default-db-registry-298031-6fdccf6498-qc68v MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "default-db-registry-298031-kube-rbac-proxy" not found 24m Warning FailedMount pod/default-db-registry-298031-6fdccf6498-qc68v MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "default-db-registry-298031-kube-rbac-proxy" not found 24m Warning Unhealthy pod/default-db-registry-298031-6fdccf6498-qc68v Startup probe failed: Get "http://10.132.0.94:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 24m Warning Unhealthy pod/default-db-registry-298031-6fdccf6498-qc68v Startup probe failed: HTTP probe failed with statuscode: 503 23m Warning FailedMount pod/test-model-registry-662874-55c7499d55-x5wb6 MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "test-model-registry-662874-kube-rbac-proxy" not found 23m Warning FailedMount pod/test-model-registry-662874-55c7499d55-x5wb6 MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "test-model-registry-662874-kube-rbac-proxy" not found 21m Warning Unhealthy pod/model-registry-db-662874-f958cd66c-wthfh Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 6m59s Warning Unhealthy pod/model-catalog-postgres-75fcbb584-bchkw Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 6m57s Warning Unhealthy pod/model-catalog-b5b55bdfb-m889w Readiness probe failed: Get "http://10.134.0.60:8080/readyz": dial tcp 10.134.0.60:8080: connect: connection refused 6m57s Warning Unhealthy pod/model-catalog-b5b55bdfb-m889w Readiness probe failed: Get "https://10.134.0.60:8888/healthz": dial tcp 10.134.0.60:8888: connect: connection refused --- namespace/rhods-notebooks --- LAST SEEN TYPE REASON OBJECT MESSAGE --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-09-09T19:15:27Z) --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-7fdc98ddcb-5rq8w.log ==================== ERROR: cannot export logs for pod redhat-ods-operator/rhods-operator-7fdc98ddcb-5rq8w: Error from server (NotFound): pods "ip-10-0-131-109.ec2.internal" not found ==================== OPERATOR POD LOG: rhods-operator-7fdc98ddcb-h9n9p.log ==================== ==================== OPERATOR POD LOG: rhods-operator-7fdc98ddcb-l8b4q.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/agent-ops-ui-687b4b5d4f-znfqj.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/ai-gateway-operator-89f69845b-998kd.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/aihub-controller-manager-d877888b8-kblb7.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/automl-ui-74d9b97d49-xwd6p.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/autorag-ui-58f548cbd-2gm2x.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/catalog-controller-manager-6b9685ffd6-vhtrv.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-operator-669d6564bd-d8vj2.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-5646fb5865-rm57x.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/data-science-pipelines-operator-controller-manager-56b8f896tfvl.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/feast-operator-controller-manager-5fc6bb8d67-h5gkp.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/feast-operator-controller-manager-5fc6bb8d67-h5gkp: Error from server (NotFound): pods "ip-10-0-131-109.ec2.internal" not found ==================== WORKLOAD POD LOG: redhat-ods-applications/gen-ai-ui-98ccc4f97-pf7h6.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kserve-module-controller-manager-66ff7c977-n9frx.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kubeflow-trainer-controller-manager-5449979c85-ggbks.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kuberay-operator-7d87f54798-6m9kt.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-controller-56c898799-d5rjs.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/maas-controller-56c898799-d5rjs: Error from server (NotFound): pods "ip-10-0-131-109.ec2.internal" not found ==================== WORKLOAD POD LOG: redhat-ods-applications/maas-ui-5bfd6fdb7d-hx8sf.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-registry-operator-controller-manager-7cd9fc56bf-jmdf6.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-registry-ui-6796658d87-8r98q.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/notebook-controller-deployment-75c8689979-wrssr.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/notebooks-ui-6577cc467-tfr8f.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-notebook-controller-manager-584b89556c-4bq7j.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/ogx-k8s-operator-controller-manager-59777cdbbb-szbgp.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/ogx-k8s-operator-controller-manager-59777cdbbb-szbgp: Error from server (NotFound): pods "ip-10-0-131-109.ec2.internal" not found ==================== WORKLOAD POD LOG: redhat-ods-applications/opendatahub-feast-operator-84ff599545-qjwnx.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/opendatahub-ogx-operator-644d45d5b-bbj2w.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/rhods-dashboard-54ff89c8f8-v68jq.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/rhods-dashboard-7764b9895c-8vjj8.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/rhods-dashboard-dd86b5b58-57khm.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-controller-c594b967f-5k6l8.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/spark-operator-controller-c594b967f-5k6l8: Error from server (NotFound): pods "ip-10-0-131-109.ec2.internal" not found ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-module-controller-manager-59549b5854-lbbcs.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/spark-operator-webhook-5d466b7d66-4zh24.log ==================== ERROR: cannot export logs for pod redhat-ods-applications/spark-operator-webhook-5d466b7d66-4zh24: Error from server (NotFound): pods "ip-10-0-131-109.ec2.internal" not found ==================== WORKLOAD POD LOG: redhat-ods-applications/trainer-operator-controller-manager-6b585cbb9f-9xz9m.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/workbenches-operator-c674d9c9-bwtk6.log ====================