==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== ResolutionFailed: status=True reason=ErrorPreventedResolution message=error using catalogsource openshift-marketplace/rhoai-catalog-dev: failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF" redhat-ods-applications/dashboard-redirect: ready=0/2 available=0 redhat-ods-applications/data-science-pipelines-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/feast-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-trainer-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-training-operator: ready=0/1 available=0 redhat-ods-applications/kuberay-operator: ready=0/1 available=0 redhat-ods-applications/llmisvc-controller-manager: ready=0/1 available=0 redhat-ods-applications/mlflow-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-registry-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-serving-api: ready=0/1 available=0 redhat-ods-applications/notebook-controller-deployment: ready=0/1 available=0 redhat-ods-applications/odh-model-controller: ready=0/1 available=0 redhat-ods-applications/odh-notebook-controller-manager: ready=0/1 available=0 redhat-ods-applications/ogx-k8s-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/rhods-dashboard: ready=0/2 available=0 redhat-ods-applications/spark-operator-controller: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook: ready=0/1 available=0 redhat-ods-applications/trustyai-service-operator-controller-manager: ready=0/1 available=0 redhat-ods-operator/rhods-operator: ready=0/3 available=0 rhoai-model-registries/model-catalog: ready=0/1 available=0 rhoai-model-registries/model-catalog-postgres: ready=0/1 available=0 9m56s Warning FailedScheduling pod/model-serving-api-5679d6b4f-gv92m 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 44m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m8s Warning FailedScheduling pod/kuberay-operator-66fb75559c-nmf45 no nodes available to schedule pods 5m12s Warning FailedScheduling pod/kuberay-operator-66fb75559c-nmf45 no nodes available to schedule pods 10m Warning FailedScheduling pod/kuberay-operator-66fb75559c-nmf45 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 44m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m55s Warning FailedScheduling pod/dashboard-redirect-765c754cf5-8fz4t 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 4m42s Warning FailedScheduling pod/dashboard-redirect-765c754cf5-8fz4t no nodes available to schedule pods 4m38s Warning FailedScheduling pod/dashboard-redirect-765c754cf5-8fz4t no nodes available to schedule pods 10m Warning FailedScheduling pod/dashboard-redirect-765c754cf5-f6vsr 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m12s Warning FailedScheduling pod/dashboard-redirect-765c754cf5-f6vsr no nodes available to schedule pods 5m8s Warning FailedScheduling pod/dashboard-redirect-765c754cf5-f6vsr no nodes available to schedule pods 44m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m8s Warning FailedScheduling pod/rhods-dashboard-676945747-fvphs no nodes available to schedule pods 28m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m12s Warning FailedScheduling pod/rhods-dashboard-676945747-fvphs no nodes available to schedule pods 10m Warning FailedScheduling pod/rhods-dashboard-676945747-fvphs 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 4m38s Warning FailedScheduling pod/odh-model-controller-65fdb9c5c-8nzrs no nodes available to schedule pods 4m38s Warning FailedScheduling pod/model-registry-operator-controller-manager-79765dbdfb-2zkbs no nodes available to schedule pods 4m42s Warning FailedScheduling pod/model-registry-operator-controller-manager-79765dbdfb-2zkbs no nodes available to schedule pods 9m54s Warning FailedScheduling pod/model-registry-operator-controller-manager-79765dbdfb-2zkbs 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 4m42s Warning FailedScheduling pod/model-serving-api-5679d6b4f-gv92m no nodes available to schedule pods 4m38s Warning FailedScheduling pod/rhods-dashboard-676945747-f6f9p no nodes available to schedule pods 9m56s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-6cc8f7bhk8mn 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 4m42s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-6cc8f7bhk8mn no nodes available to schedule pods 4m38s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-6cc8f7bhk8mn no nodes available to schedule pods 4m42s Warning FailedScheduling pod/odh-model-controller-65fdb9c5c-8nzrs no nodes available to schedule pods 4m38s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-cb88d97c9-szgbg no nodes available to schedule pods 4m42s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-cb88d97c9-szgbg no nodes available to schedule pods 4m42s Warning FailedScheduling pod/feast-operator-controller-manager-7b7cd769f7-42gxb no nodes available to schedule pods 4m38s Warning FailedScheduling pod/feast-operator-controller-manager-7b7cd769f7-42gxb no nodes available to schedule pods 5m8s Warning FailedScheduling pod/mlflow-operator-controller-manager-b5455d498-k8dj2 no nodes available to schedule pods 5m12s Warning FailedScheduling pod/mlflow-operator-controller-manager-b5455d498-k8dj2 no nodes available to schedule pods 10m Warning FailedScheduling pod/mlflow-operator-controller-manager-b5455d498-k8dj2 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 4m38s Warning FailedScheduling pod/model-serving-api-5679d6b4f-gv92m no nodes available to schedule pods 4m42s Warning FailedScheduling pod/rhods-dashboard-676945747-f6f9p no nodes available to schedule pods 9m56s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-cb88d97c9-szgbg 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 9m55s Warning FailedScheduling pod/rhods-dashboard-676945747-f6f9p 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m56s Warning FailedScheduling pod/odh-model-controller-65fdb9c5c-8nzrs 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m8s Warning FailedScheduling pod/odh-notebook-controller-manager-598c78b776-hg8fl no nodes available to schedule pods 5m12s Warning FailedScheduling pod/odh-notebook-controller-manager-598c78b776-hg8fl no nodes available to schedule pods 10m Warning FailedScheduling pod/odh-notebook-controller-manager-598c78b776-hg8fl 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 9m55s Warning FailedScheduling pod/feast-operator-controller-manager-7b7cd769f7-42gxb 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 58m Warning FailedMount pod/model-registry-operator-controller-manager-79765dbdfb-s92x9 MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 58m Warning FailedMount pod/mlflow-operator-controller-manager-b5455d498-9qb67 MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 58m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-6cc8f7b6lmgm MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 58m Warning FailedMount pod/odh-model-controller-65fdb9c5c-prpqg MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 58m Warning FailedMount pod/odh-notebook-controller-manager-598c78b776-cpblq MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 58m Warning FailedCreate replicaset/notebook-controller-deployment-7c5c76fdc6 Error creating: pods "notebook-controller-deployment-7c5c76fdc6-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 58m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 58m Warning FailedCreate replicaset/rhods-dashboard-676945747 Error creating: pods "rhods-dashboard-676945747-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 58m Warning FailedMount pod/kuberay-operator-66fb75559c-r8pv9 MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 58m Warning FailedMount pod/rhods-dashboard-676945747-r25sb MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 58m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-5dccbd95db Error creating: pods "trustyai-service-operator-controller-manager-5dccbd95db-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 58m Warning FailedMount pod/trustyai-service-operator-controller-manager-5dccbd95db-sxjcm MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 44m Warning FailedMount pod/ogx-k8s-operator-controller-manager-5b84b76f79-vzljm MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 44m Warning FailedCreate replicaset/spark-operator-controller-5dd7d48f87 Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 12m Warning Unhealthy pod/rhods-dashboard-676945747-r25sb Readiness probe failed: 12m Warning Unhealthy pod/spark-operator-controller-5dd7d48f87-cmcwg Readiness probe failed: Get "http://10.133.0.53:8081/readyz": dial tcp 10.133.0.53:8081: connect: connection refused 11m Warning NodeNotReady pod/spark-operator-webhook-5f8768dbfd-jxn79 Node is not ready 11m Warning NodeNotReady pod/mlflow-operator-controller-manager-b5455d498-9qb67 Node is not ready 11m Warning NodeNotReady pod/llmisvc-controller-manager-8685b58fdb-vlzpz Node is not ready 11m Warning NodeNotReady pod/notebook-controller-deployment-7c5c76fdc6-b8rlc Node is not ready 11m Warning NodeNotReady pod/ogx-k8s-operator-controller-manager-5b84b76f79-vzljm Node is not ready 11m Warning NodeNotReady pod/odh-notebook-controller-manager-598c78b776-cpblq Node is not ready 11m Warning NodeNotReady pod/dashboard-redirect-765c754cf5-x9c6w Node is not ready 11m Warning NodeNotReady pod/kserve-controller-manager-db7bd7757-qj4w4 Node is not ready 11m Warning NodeNotReady pod/rhods-dashboard-676945747-mhw6j Node is not ready 11m Warning NodeNotReady pod/kuberay-operator-66fb75559c-r8pv9 Node is not ready 11m Warning NodeNotReady pod/kubeflow-training-operator-54786944d6-87r7n Node is not ready 11m Warning NodeNotReady pod/spark-operator-controller-5dd7d48f87-cmcwg Node is not ready 11m Warning NodeNotReady pod/trustyai-service-operator-controller-manager-5dccbd95db-sxjcm Node is not ready 11m Warning NodeNotReady pod/rhods-dashboard-676945747-r25sb Node is not ready 11m Warning NodeNotReady pod/model-registry-operator-controller-manager-79765dbdfb-s92x9 Node is not ready 11m Warning NodeNotReady pod/data-science-pipelines-operator-controller-manager-6cc8f7b6lmgm Node is not ready 11m Warning NodeNotReady pod/model-serving-api-5679d6b4f-sh8fb Node is not ready 11m Warning NodeNotReady pod/odh-model-controller-65fdb9c5c-prpqg Node is not ready 11m Warning NodeNotReady pod/dashboard-redirect-765c754cf5-86ldj Node is not ready 11m Warning NodeNotReady pod/feast-operator-controller-manager-7b7cd769f7-g9z6j Node is not ready 11m Warning NodeNotReady pod/kubeflow-trainer-controller-manager-cb88d97c9-r685n Node is not ready 10m Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: skipping Pod rhods-dashboard-676945747-r25sb for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-131-94.ec2.internal Not Found 10m Warning FailedToUpdateEndpointSlices service/dashboard-redirect Error updating Endpoint Slices for Service redhat-ods-applications/dashboard-redirect: skipping Pod dashboard-redirect-765c754cf5-86ldj for Service redhat-ods-applications/dashboard-redirect: Node ip-10-0-131-94.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/kubeflow-training-operator Error updating Endpoint Slices for Service redhat-ods-applications/kubeflow-training-operator: skipping Pod kubeflow-training-operator-54786944d6-87r7n for Service redhat-ods-applications/kubeflow-training-operator: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/ogx-k8s-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-service: skipping Pod ogx-k8s-operator-controller-manager-5b84b76f79-vzljm for Service redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/trustyai-service-operator-webhook-service Error updating Endpoint Slices for Service redhat-ods-applications/trustyai-service-operator-webhook-service: skipping Pod trustyai-service-operator-controller-manager-5dccbd95db-sxjcm for Service redhat-ods-applications/trustyai-service-operator-webhook-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/trustyai-service-operator-metrics-service Error updating Endpoint Slices for Service redhat-ods-applications/trustyai-service-operator-metrics-service: skipping Pod trustyai-service-operator-controller-manager-5dccbd95db-sxjcm for Service redhat-ods-applications/trustyai-service-operator-metrics-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/spark-operator-webhook-svc Error updating Endpoint Slices for Service redhat-ods-applications/spark-operator-webhook-svc: skipping Pod spark-operator-webhook-5f8768dbfd-jxn79 for Service redhat-ods-applications/spark-operator-webhook-svc: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/trustyai-service-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-applications/trustyai-service-operator-controller-manager-metrics-service: skipping Pod trustyai-service-operator-controller-manager-5dccbd95db-sxjcm for Service redhat-ods-applications/trustyai-service-operator-controller-manager-metrics-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/notebook-controller-service Error updating Endpoint Slices for Service redhat-ods-applications/notebook-controller-service: skipping Pod notebook-controller-deployment-7c5c76fdc6-b8rlc for Service redhat-ods-applications/notebook-controller-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/ogx-k8s-operator-webhook-service Error updating Endpoint Slices for Service redhat-ods-applications/ogx-k8s-operator-webhook-service: skipping Pod ogx-k8s-operator-controller-manager-5b84b76f79-vzljm for Service redhat-ods-applications/ogx-k8s-operator-webhook-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/kserve-webhook-server-service Error updating Endpoint Slices for Service redhat-ods-applications/kserve-webhook-server-service: skipping Pod kserve-controller-manager-db7bd7757-qj4w4 for Service redhat-ods-applications/kserve-webhook-server-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/kserve-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-applications/kserve-controller-manager-metrics-service: skipping Pod kserve-controller-manager-db7bd7757-qj4w4 for Service redhat-ods-applications/kserve-controller-manager-metrics-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/llmisvc-controller-manager-service Error updating Endpoint Slices for Service redhat-ods-applications/llmisvc-controller-manager-service: skipping Pod llmisvc-controller-manager-8685b58fdb-vlzpz for Service redhat-ods-applications/llmisvc-controller-manager-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/kserve-controller-manager-service Error updating Endpoint Slices for Service redhat-ods-applications/kserve-controller-manager-service: skipping Pod kserve-controller-manager-db7bd7757-qj4w4 for Service redhat-ods-applications/kserve-controller-manager-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/llmisvc-webhook-server-service Error updating Endpoint Slices for Service redhat-ods-applications/llmisvc-webhook-server-service: skipping Pod llmisvc-controller-manager-8685b58fdb-vlzpz for Service redhat-ods-applications/llmisvc-webhook-server-service: Node ip-10-0-134-204.ec2.internal Not Found 2m5s Warning FailedCreate replicaset/llmisvc-controller-manager-8685b58fdb Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m5s Warning FailedCreate replicaset/spark-operator-controller-5dd7d48f87 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m4s Warning FailedCreate replicaset/kserve-controller-manager-db7bd7757 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m3s Warning FailedCreate replicaset/ogx-k8s-operator-controller-manager-5b84b76f79 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m2s Warning FailedCreate replicaset/spark-operator-webhook-5f8768dbfd Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 115s Warning FailedCreate replicaset/notebook-controller-deployment-7c5c76fdc6 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 114s Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-5dccbd95db Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 111s Warning FailedCreate replicaset/kubeflow-training-operator-54786944d6 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 5m8s Warning FailedScheduling pod/rhods-operator-568d65fbfc-vklq9 no nodes available to schedule pods 4m38s Warning FailedScheduling pod/rhods-operator-568d65fbfc-zxp47 no nodes available to schedule pods 4m43s Warning FailedScheduling pod/rhods-operator-568d65fbfc-zxp47 no nodes available to schedule pods 9m57s Warning FailedScheduling pod/rhods-operator-568d65fbfc-zxp47 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m13s Warning FailedScheduling pod/rhods-operator-568d65fbfc-vklq9 no nodes available to schedule pods 10m Warning FailedScheduling pod/rhods-operator-568d65fbfc-vklq9 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 59m Warning AppliedWithWarnings installplan/install-mrqks 1 warning(s) generated during installation of operator "rhods-operator.3.5.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 11m Warning NodeNotReady pod/rhods-operator-568d65fbfc-k6vpc Node is not ready 11m Warning NodeNotReady pod/rhods-operator-568d65fbfc-kxjzb Node is not ready 11m Warning ComponentUnhealthy clusterserviceversion/rhods-operator.3.5.0-ea.2 installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. 11m Warning NodeNotReady pod/rhods-operator-568d65fbfc-8j8q4 Node is not ready 10m Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: skipping Pod rhods-operator-568d65fbfc-8j8q4 for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-131-94.ec2.internal Not Found 10m Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: skipping Pod rhods-operator-568d65fbfc-8j8q4 for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-131-94.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: skipping Pod rhods-operator-568d65fbfc-k6vpc for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: skipping Pod rhods-operator-568d65fbfc-k6vpc for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-134-204.ec2.internal Not Found 4m36s Warning FailedCreate replicaset/rhods-operator-568d65fbfc Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 88s Warning InstallCheckFailed clusterserviceversion/rhods-operator.3.5.0-ea.2 install timeout 4m42s Warning FailedScheduling pod/model-catalog-6dcf85dd5d-ld442 no nodes available to schedule pods 30m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 31m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 31m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 30m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 9m55s Warning FailedScheduling pod/model-catalog-6dcf85dd5d-ld442 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 4m38s Warning FailedScheduling pod/model-catalog-6dcf85dd5d-ld442 no nodes available to schedule pods 58m Warning FailedMount pod/model-catalog-6dcf85dd5d-zfbkl MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 58m Warning Unhealthy pod/model-catalog-6dcf85dd5d-zfbkl Readiness probe failed: Get "http://10.132.0.57:8080/readyz": dial tcp 10.132.0.57:8080: connect: connection refused 31m Warning Unhealthy pod/db-model-registry0-865b4f5d6f-z6dgm Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 11m Warning NodeNotReady pod/model-catalog-postgres-566f8bdddc-b5qvp Node is not ready 11m Warning NodeNotReady pod/model-catalog-6dcf85dd5d-zfbkl Node is not ready 8m24s Warning FailedToUpdateEndpointSlices service/model-catalog-postgres Error updating Endpoint Slices for Service rhoai-model-registries/model-catalog-postgres: skipping Pod model-catalog-postgres-566f8bdddc-b5qvp for Service rhoai-model-registries/model-catalog-postgres: Node ip-10-0-134-204.ec2.internal Not Found 3m2s Warning FailedCreate replicaset/model-catalog-postgres-566f8bdddc Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found (no matching error/webhook lines in operator logs for this time window) --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-07-14T10:37:52Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.21.24 Kubernetes Version: v1.34.9 --- Namespaces --- namespace/opendatahub namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=True reason=UnhealthyCatalogSourceFound message=targeted catalogsource openshift-marketplace/rhoai-catalog-dev unhealthy ResolutionFailed: status=True reason=ErrorPreventedResolution message=error using catalogsource openshift-marketplace/rhoai-catalog-dev: failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF" --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.1 Authorino Operator 1.4.1 authorino-operator.v1.3.2 Installing cert-manager-operator.v1.20.0 cert-manager Operator for Red Hat OpenShift 1.20.0 cert-manager-operator.v1.19.0 Installing custom-metrics-autoscaler.v2.19.0-2 Custom Metrics Autoscaler 2.19.0-2 custom-metrics-autoscaler.v2.19.0-1 Installing dns-operator.v1.4.0 DNS Operator 1.4.0 dns-operator.v1.3.1 Installing kueue-operator.v1.3.1 Red Hat build of Kueue 1.3.1 Installing limitador-operator.v1.4.0 Limitador Operator 1.4.0 limitador-operator.v1.3.1 Installing rhcl-operator.v1.4.1 Red Hat Connectivity Link 1.4.1 rhcl-operator.v1.3.5 Installing rhods-operator.3.5.0-ea.2 Red Hat OpenShift AI 3.5.0-ea.2 Installing --- CSVs not in Succeeded phase --- authorino-operator.v1.4.1: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace cert-manager-operator.v1.20.0: phase=Installing reason=Copied message=The operator is running in cert-manager-operator but is managing this namespace custom-metrics-autoscaler.v2.19.0-2: phase=Installing reason=Copied message=The operator is running in openshift-keda but is managing this namespace dns-operator.v1.4.0: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace kueue-operator.v1.3.1: phase=Installing reason=Copied message=The operator is running in openshift-kueue-operator but is managing this namespace limitador-operator.v1.4.0: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace rhcl-operator.v1.4.1: phase=Installing reason=Copied message=The operator is running in kuadrant-system but is managing this namespace rhods-operator.3.5.0-ea.2: phase=Installing reason=InstallWaiting message=installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aigateway: managementState=n/a aipipelines: managementState=Managed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Removed llamastackoperator: managementState=n/a mlflowoperator: managementState=Managed modelregistry: managementState=Managed ogx: managementState=Managed ray: managementState=Managed sparkoperator: managementState=Managed trainer: managementState=Managed trainingoperator: managementState=Managed trustyai: managementState=Managed workbenches: managementState=Managed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= AIPipelinesReady: status=True reason= message= DashboardReady: status=True reason= message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=False reason=Removed message=Component ManagementState is set to Removed LLM-D-WVADependencies: status=True reason= message= MLflowOperatorReady: status=True reason= message= ModelControllerReady: status=True reason= message= ModelRegistryReady: status=True reason= message= ModelsAsServiceReady: status=False reason=Removed message=Component ManagementState is set to Removed OGXReady: status=True reason= message= RayReady: status=True reason= message= SparkOperatorReady: status=True reason= message= TrainerReady: status=True reason= message= TrainingOperatorReady: status=True reason= message= TrustyAIReady: status=True reason= message= WorkbenchesReady: status=True reason= message= --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 58m Ready 2026-07-14T09:39:03Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=True reason=Ready message=Monitoring stack is initialized Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in DSCI CR OpenTelemetryCollectorAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Metrics not configured in DSCI CR Traces not configured in DSCI CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in DSCI CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR PersesAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR --- Component CRs (Ready) --- NAME READY REASON URL dashboard.components.platform.opendatahub.io/default-dashboard True NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON mlflowoperator.components.platform.opendatahub.io/default-mlflowoperator True NAME READY REASON URI modelcontroller.components.platform.opendatahub.io/default-modelcontroller True NAME READY REASON modelregistry.components.platform.opendatahub.io/default-modelregistry True NAME READY REASON ogx.components.platform.opendatahub.io/default-ogx True NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON sparkoperator.components.platform.opendatahub.io/default-sparkoperator True NAME READY REASON trainer.components.platform.opendatahub.io/default-trainer True NAME READY REASON trainingoperator.components.platform.opendatahub.io/default-trainingoperator True NAME READY REASON trustyai.components.platform.opendatahub.io/default-trustyai True NAME READY REASON workbenches.components.platform.opendatahub.io/default-workbenches True --- Deployments not fully ready (ODH/RHOAI namespaces) --- redhat-ods-applications/dashboard-redirect: ready=0/2 available=0 redhat-ods-applications/data-science-pipelines-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/feast-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/kserve-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-trainer-controller-manager: ready=0/1 available=0 redhat-ods-applications/kubeflow-training-operator: ready=0/1 available=0 redhat-ods-applications/kuberay-operator: ready=0/1 available=0 redhat-ods-applications/llmisvc-controller-manager: ready=0/1 available=0 redhat-ods-applications/mlflow-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-registry-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/model-serving-api: ready=0/1 available=0 redhat-ods-applications/notebook-controller-deployment: ready=0/1 available=0 redhat-ods-applications/odh-model-controller: ready=0/1 available=0 redhat-ods-applications/odh-notebook-controller-manager: ready=0/1 available=0 redhat-ods-applications/ogx-k8s-operator-controller-manager: ready=0/1 available=0 redhat-ods-applications/rhods-dashboard: ready=0/2 available=0 redhat-ods-applications/spark-operator-controller: ready=0/1 available=0 redhat-ods-applications/spark-operator-webhook: ready=0/1 available=0 redhat-ods-applications/trustyai-service-operator-controller-manager: ready=0/1 available=0 redhat-ods-operator/rhods-operator: ready=0/3 available=0 rhoai-model-registries/model-catalog: ready=0/1 available=0 rhoai-model-registries/model-catalog-postgres: ready=0/1 available=0 --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- redhat-ods-applications/dashboard-redirect-765c754cf5-8fz4t: phase=Pending reason= redhat-ods-applications/dashboard-redirect-765c754cf5-f6vsr: phase=Pending reason= redhat-ods-applications/data-science-pipelines-operator-controller-manager-6cc8f7bhk8mn: phase=Pending reason= redhat-ods-applications/feast-operator-controller-manager-7b7cd769f7-42gxb: phase=Pending reason= redhat-ods-applications/kubeflow-trainer-controller-manager-cb88d97c9-szgbg: phase=Pending reason= redhat-ods-applications/kuberay-operator-66fb75559c-nmf45: phase=Pending reason= redhat-ods-applications/mlflow-operator-controller-manager-b5455d498-k8dj2: phase=Pending reason= redhat-ods-applications/model-registry-operator-controller-manager-79765dbdfb-2zkbs: phase=Pending reason= redhat-ods-applications/model-serving-api-5679d6b4f-gv92m: phase=Pending reason= redhat-ods-applications/odh-model-controller-65fdb9c5c-8nzrs: phase=Pending reason= redhat-ods-applications/odh-notebook-controller-manager-598c78b776-hg8fl: phase=Pending reason= redhat-ods-applications/rhods-dashboard-676945747-f6f9p: phase=Pending reason= redhat-ods-applications/rhods-dashboard-676945747-fvphs: phase=Pending reason= redhat-ods-operator/rhods-operator-568d65fbfc-vklq9: phase=Pending reason= redhat-ods-operator/rhods-operator-568d65fbfc-zxp47: phase=Pending reason= rhoai-model-registries/model-catalog-6dcf85dd5d-ld442: phase=Pending reason= --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/opendatahub --- (no events) --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 9m56s Warning FailedScheduling pod/model-serving-api-5679d6b4f-gv92m 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 44m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 28m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m8s Warning FailedScheduling pod/kuberay-operator-66fb75559c-nmf45 no nodes available to schedule pods 5m12s Warning FailedScheduling pod/kuberay-operator-66fb75559c-nmf45 no nodes available to schedule pods 10m Warning FailedScheduling pod/kuberay-operator-66fb75559c-nmf45 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 44m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m55s Warning FailedScheduling pod/dashboard-redirect-765c754cf5-8fz4t 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 4m42s Warning FailedScheduling pod/dashboard-redirect-765c754cf5-8fz4t no nodes available to schedule pods 4m38s Warning FailedScheduling pod/dashboard-redirect-765c754cf5-8fz4t no nodes available to schedule pods 10m Warning FailedScheduling pod/dashboard-redirect-765c754cf5-f6vsr 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m12s Warning FailedScheduling pod/dashboard-redirect-765c754cf5-f6vsr no nodes available to schedule pods 5m8s Warning FailedScheduling pod/dashboard-redirect-765c754cf5-f6vsr no nodes available to schedule pods 44m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 44m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 43m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m8s Warning FailedScheduling pod/rhods-dashboard-676945747-fvphs no nodes available to schedule pods 28m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 5m12s Warning FailedScheduling pod/rhods-dashboard-676945747-fvphs no nodes available to schedule pods 10m Warning FailedScheduling pod/rhods-dashboard-676945747-fvphs 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 4m38s Warning FailedScheduling pod/odh-model-controller-65fdb9c5c-8nzrs no nodes available to schedule pods 4m38s Warning FailedScheduling pod/model-registry-operator-controller-manager-79765dbdfb-2zkbs no nodes available to schedule pods 4m42s Warning FailedScheduling pod/model-registry-operator-controller-manager-79765dbdfb-2zkbs no nodes available to schedule pods 9m54s Warning FailedScheduling pod/model-registry-operator-controller-manager-79765dbdfb-2zkbs 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 4m42s Warning FailedScheduling pod/model-serving-api-5679d6b4f-gv92m no nodes available to schedule pods 4m38s Warning FailedScheduling pod/rhods-dashboard-676945747-f6f9p no nodes available to schedule pods 9m56s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-6cc8f7bhk8mn 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 4m42s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-6cc8f7bhk8mn no nodes available to schedule pods 4m38s Warning FailedScheduling pod/data-science-pipelines-operator-controller-manager-6cc8f7bhk8mn no nodes available to schedule pods 4m42s Warning FailedScheduling pod/odh-model-controller-65fdb9c5c-8nzrs no nodes available to schedule pods 4m38s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-cb88d97c9-szgbg no nodes available to schedule pods 4m42s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-cb88d97c9-szgbg no nodes available to schedule pods 4m42s Warning FailedScheduling pod/feast-operator-controller-manager-7b7cd769f7-42gxb no nodes available to schedule pods 4m38s Warning FailedScheduling pod/feast-operator-controller-manager-7b7cd769f7-42gxb no nodes available to schedule pods 5m8s Warning FailedScheduling pod/mlflow-operator-controller-manager-b5455d498-k8dj2 no nodes available to schedule pods 5m12s Warning FailedScheduling pod/mlflow-operator-controller-manager-b5455d498-k8dj2 no nodes available to schedule pods 10m Warning FailedScheduling pod/mlflow-operator-controller-manager-b5455d498-k8dj2 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 4m38s Warning FailedScheduling pod/model-serving-api-5679d6b4f-gv92m no nodes available to schedule pods 4m42s Warning FailedScheduling pod/rhods-dashboard-676945747-f6f9p no nodes available to schedule pods 9m56s Warning FailedScheduling pod/kubeflow-trainer-controller-manager-cb88d97c9-szgbg 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 9m55s Warning FailedScheduling pod/rhods-dashboard-676945747-f6f9p 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 20m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 9m56s Warning FailedScheduling pod/odh-model-controller-65fdb9c5c-8nzrs 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m8s Warning FailedScheduling pod/odh-notebook-controller-manager-598c78b776-hg8fl no nodes available to schedule pods 5m12s Warning FailedScheduling pod/odh-notebook-controller-manager-598c78b776-hg8fl no nodes available to schedule pods 10m Warning FailedScheduling pod/odh-notebook-controller-manager-598c78b776-hg8fl 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 9m55s Warning FailedScheduling pod/feast-operator-controller-manager-7b7cd769f7-42gxb 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 58m Warning FailedMount pod/model-registry-operator-controller-manager-79765dbdfb-s92x9 MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 58m Warning FailedMount pod/mlflow-operator-controller-manager-b5455d498-9qb67 MountVolume.SetUp failed for volume "metrics-cert" : secret "controller-manager-metrics-tls" not found 58m Warning FailedMount pod/data-science-pipelines-operator-controller-manager-6cc8f7b6lmgm MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 58m Warning FailedMount pod/odh-model-controller-65fdb9c5c-prpqg MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 58m Warning FailedMount pod/odh-notebook-controller-manager-598c78b776-cpblq MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 58m Warning FailedCreate replicaset/notebook-controller-deployment-7c5c76fdc6 Error creating: pods "notebook-controller-deployment-7c5c76fdc6-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 58m Warning FailedCreate replicaset/model-serving-api-5679d6b4f Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 58m Warning FailedCreate replicaset/rhods-dashboard-676945747 Error creating: pods "rhods-dashboard-676945747-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 58m Warning FailedMount pod/kuberay-operator-66fb75559c-r8pv9 MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 58m Warning FailedMount pod/rhods-dashboard-676945747-r25sb MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 58m Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-5dccbd95db Error creating: pods "trustyai-service-operator-controller-manager-5dccbd95db-" is forbidden: error looking up service account redhat-ods-applications/trustyai-service-operator-controller-manager: serviceaccount "trustyai-service-operator-controller-manager" not found 58m Warning FailedMount pod/trustyai-service-operator-controller-manager-5dccbd95db-sxjcm MountVolume.SetUp failed for volume "webhook-certs" : secret "webhook-server-cert" not found 44m Warning FailedMount pod/ogx-k8s-operator-controller-manager-5b84b76f79-vzljm MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 44m Warning FailedCreate replicaset/spark-operator-controller-5dd7d48f87 Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 12m Warning Unhealthy pod/rhods-dashboard-676945747-r25sb Readiness probe failed: 12m Warning Unhealthy pod/spark-operator-controller-5dd7d48f87-cmcwg Readiness probe failed: Get "http://10.133.0.53:8081/readyz": dial tcp 10.133.0.53:8081: connect: connection refused 11m Warning NodeNotReady pod/spark-operator-webhook-5f8768dbfd-jxn79 Node is not ready 11m Warning NodeNotReady pod/mlflow-operator-controller-manager-b5455d498-9qb67 Node is not ready 11m Warning NodeNotReady pod/llmisvc-controller-manager-8685b58fdb-vlzpz Node is not ready 11m Warning NodeNotReady pod/notebook-controller-deployment-7c5c76fdc6-b8rlc Node is not ready 11m Warning NodeNotReady pod/ogx-k8s-operator-controller-manager-5b84b76f79-vzljm Node is not ready 11m Warning NodeNotReady pod/odh-notebook-controller-manager-598c78b776-cpblq Node is not ready 11m Warning NodeNotReady pod/dashboard-redirect-765c754cf5-x9c6w Node is not ready 11m Warning NodeNotReady pod/kserve-controller-manager-db7bd7757-qj4w4 Node is not ready 11m Warning NodeNotReady pod/rhods-dashboard-676945747-mhw6j Node is not ready 11m Warning NodeNotReady pod/kuberay-operator-66fb75559c-r8pv9 Node is not ready 11m Warning NodeNotReady pod/kubeflow-training-operator-54786944d6-87r7n Node is not ready 11m Warning NodeNotReady pod/spark-operator-controller-5dd7d48f87-cmcwg Node is not ready 11m Warning NodeNotReady pod/trustyai-service-operator-controller-manager-5dccbd95db-sxjcm Node is not ready 11m Warning NodeNotReady pod/rhods-dashboard-676945747-r25sb Node is not ready 11m Warning NodeNotReady pod/model-registry-operator-controller-manager-79765dbdfb-s92x9 Node is not ready 11m Warning NodeNotReady pod/data-science-pipelines-operator-controller-manager-6cc8f7b6lmgm Node is not ready 11m Warning NodeNotReady pod/model-serving-api-5679d6b4f-sh8fb Node is not ready 11m Warning NodeNotReady pod/odh-model-controller-65fdb9c5c-prpqg Node is not ready 11m Warning NodeNotReady pod/dashboard-redirect-765c754cf5-86ldj Node is not ready 11m Warning NodeNotReady pod/feast-operator-controller-manager-7b7cd769f7-g9z6j Node is not ready 11m Warning NodeNotReady pod/kubeflow-trainer-controller-manager-cb88d97c9-r685n Node is not ready 10m Warning FailedToUpdateEndpointSlices service/rhods-dashboard Error updating Endpoint Slices for Service redhat-ods-applications/rhods-dashboard: skipping Pod rhods-dashboard-676945747-r25sb for Service redhat-ods-applications/rhods-dashboard: Node ip-10-0-131-94.ec2.internal Not Found 10m Warning FailedToUpdateEndpointSlices service/dashboard-redirect Error updating Endpoint Slices for Service redhat-ods-applications/dashboard-redirect: skipping Pod dashboard-redirect-765c754cf5-86ldj for Service redhat-ods-applications/dashboard-redirect: Node ip-10-0-131-94.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/kubeflow-training-operator Error updating Endpoint Slices for Service redhat-ods-applications/kubeflow-training-operator: skipping Pod kubeflow-training-operator-54786944d6-87r7n for Service redhat-ods-applications/kubeflow-training-operator: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/ogx-k8s-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-service: skipping Pod ogx-k8s-operator-controller-manager-5b84b76f79-vzljm for Service redhat-ods-applications/ogx-k8s-operator-controller-manager-metrics-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/trustyai-service-operator-webhook-service Error updating Endpoint Slices for Service redhat-ods-applications/trustyai-service-operator-webhook-service: skipping Pod trustyai-service-operator-controller-manager-5dccbd95db-sxjcm for Service redhat-ods-applications/trustyai-service-operator-webhook-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/trustyai-service-operator-metrics-service Error updating Endpoint Slices for Service redhat-ods-applications/trustyai-service-operator-metrics-service: skipping Pod trustyai-service-operator-controller-manager-5dccbd95db-sxjcm for Service redhat-ods-applications/trustyai-service-operator-metrics-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/spark-operator-webhook-svc Error updating Endpoint Slices for Service redhat-ods-applications/spark-operator-webhook-svc: skipping Pod spark-operator-webhook-5f8768dbfd-jxn79 for Service redhat-ods-applications/spark-operator-webhook-svc: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/trustyai-service-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-applications/trustyai-service-operator-controller-manager-metrics-service: skipping Pod trustyai-service-operator-controller-manager-5dccbd95db-sxjcm for Service redhat-ods-applications/trustyai-service-operator-controller-manager-metrics-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/notebook-controller-service Error updating Endpoint Slices for Service redhat-ods-applications/notebook-controller-service: skipping Pod notebook-controller-deployment-7c5c76fdc6-b8rlc for Service redhat-ods-applications/notebook-controller-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/ogx-k8s-operator-webhook-service Error updating Endpoint Slices for Service redhat-ods-applications/ogx-k8s-operator-webhook-service: skipping Pod ogx-k8s-operator-controller-manager-5b84b76f79-vzljm for Service redhat-ods-applications/ogx-k8s-operator-webhook-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/kserve-webhook-server-service Error updating Endpoint Slices for Service redhat-ods-applications/kserve-webhook-server-service: skipping Pod kserve-controller-manager-db7bd7757-qj4w4 for Service redhat-ods-applications/kserve-webhook-server-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/kserve-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-applications/kserve-controller-manager-metrics-service: skipping Pod kserve-controller-manager-db7bd7757-qj4w4 for Service redhat-ods-applications/kserve-controller-manager-metrics-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/llmisvc-controller-manager-service Error updating Endpoint Slices for Service redhat-ods-applications/llmisvc-controller-manager-service: skipping Pod llmisvc-controller-manager-8685b58fdb-vlzpz for Service redhat-ods-applications/llmisvc-controller-manager-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/kserve-controller-manager-service Error updating Endpoint Slices for Service redhat-ods-applications/kserve-controller-manager-service: skipping Pod kserve-controller-manager-db7bd7757-qj4w4 for Service redhat-ods-applications/kserve-controller-manager-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/llmisvc-webhook-server-service Error updating Endpoint Slices for Service redhat-ods-applications/llmisvc-webhook-server-service: skipping Pod llmisvc-controller-manager-8685b58fdb-vlzpz for Service redhat-ods-applications/llmisvc-webhook-server-service: Node ip-10-0-134-204.ec2.internal Not Found 2m5s Warning FailedCreate replicaset/llmisvc-controller-manager-8685b58fdb Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m5s Warning FailedCreate replicaset/spark-operator-controller-5dd7d48f87 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m4s Warning FailedCreate replicaset/kserve-controller-manager-db7bd7757 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m3s Warning FailedCreate replicaset/ogx-k8s-operator-controller-manager-5b84b76f79 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2m2s Warning FailedCreate replicaset/spark-operator-webhook-5f8768dbfd Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 115s Warning FailedCreate replicaset/notebook-controller-deployment-7c5c76fdc6 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 114s Warning FailedCreate replicaset/trustyai-service-operator-controller-manager-5dccbd95db Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 111s Warning FailedCreate replicaset/kubeflow-training-operator-54786944d6 Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- LAST SEEN TYPE REASON OBJECT MESSAGE 5m8s Warning FailedScheduling pod/rhods-operator-568d65fbfc-vklq9 no nodes available to schedule pods 4m38s Warning FailedScheduling pod/rhods-operator-568d65fbfc-zxp47 no nodes available to schedule pods 4m43s Warning FailedScheduling pod/rhods-operator-568d65fbfc-zxp47 no nodes available to schedule pods 9m57s Warning FailedScheduling pod/rhods-operator-568d65fbfc-zxp47 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 5m13s Warning FailedScheduling pod/rhods-operator-568d65fbfc-vklq9 no nodes available to schedule pods 10m Warning FailedScheduling pod/rhods-operator-568d65fbfc-vklq9 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 59m Warning AppliedWithWarnings installplan/install-mrqks 1 warning(s) generated during installation of operator "rhods-operator.3.5.0-ea.2" (CustomResourceDefinition "hardwareprofiles.infrastructure.opendatahub.io"): infrastructure.opendatahub.io/v1alpha1 is deprecated; please use infrastructure.opendatahub.io/v1 11m Warning NodeNotReady pod/rhods-operator-568d65fbfc-k6vpc Node is not ready 11m Warning NodeNotReady pod/rhods-operator-568d65fbfc-kxjzb Node is not ready 11m Warning ComponentUnhealthy clusterserviceversion/rhods-operator.3.5.0-ea.2 installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability. 11m Warning NodeNotReady pod/rhods-operator-568d65fbfc-8j8q4 Node is not ready 10m Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: skipping Pod rhods-operator-568d65fbfc-8j8q4 for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-131-94.ec2.internal Not Found 10m Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: skipping Pod rhods-operator-568d65fbfc-8j8q4 for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-131-94.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/redhat-ods-operator-controller-manager-metrics-service Error updating Endpoint Slices for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: skipping Pod rhods-operator-568d65fbfc-k6vpc for Service redhat-ods-operator/redhat-ods-operator-controller-manager-metrics-service: Node ip-10-0-134-204.ec2.internal Not Found 7m48s Warning FailedToUpdateEndpointSlices service/rhods-operator-service Error updating Endpoint Slices for Service redhat-ods-operator/rhods-operator-service: skipping Pod rhods-operator-568d65fbfc-k6vpc for Service redhat-ods-operator/rhods-operator-service: Node ip-10-0-134-204.ec2.internal Not Found 4m36s Warning FailedCreate replicaset/rhods-operator-568d65fbfc Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 88s Warning InstallCheckFailed clusterserviceversion/rhods-operator.3.5.0-ea.2 install timeout --- namespace/rhoai-model-registries --- LAST SEEN TYPE REASON OBJECT MESSAGE 4m42s Warning FailedScheduling pod/model-catalog-6dcf85dd5d-ld442 no nodes available to schedule pods 30m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 31m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 31m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 30m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 9m55s Warning FailedScheduling pod/model-catalog-6dcf85dd5d-ld442 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 4m38s Warning FailedScheduling pod/model-catalog-6dcf85dd5d-ld442 no nodes available to schedule pods 58m Warning FailedMount pod/model-catalog-6dcf85dd5d-zfbkl MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "model-catalog-kube-rbac-proxy-config" not found 58m Warning Unhealthy pod/model-catalog-6dcf85dd5d-zfbkl Readiness probe failed: Get "http://10.132.0.57:8080/readyz": dial tcp 10.132.0.57:8080: connect: connection refused 31m Warning Unhealthy pod/db-model-registry0-865b4f5d6f-z6dgm Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 11m Warning NodeNotReady pod/model-catalog-postgres-566f8bdddc-b5qvp Node is not ready 11m Warning NodeNotReady pod/model-catalog-6dcf85dd5d-zfbkl Node is not ready 8m24s Warning FailedToUpdateEndpointSlices service/model-catalog-postgres Error updating Endpoint Slices for Service rhoai-model-registries/model-catalog-postgres: skipping Pod model-catalog-postgres-566f8bdddc-b5qvp for Service rhoai-model-registries/model-catalog-postgres: Node ip-10-0-134-204.ec2.internal Not Found 3m2s Warning FailedCreate replicaset/model-catalog-postgres-566f8bdddc Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found --- namespace/rhods-notebooks --- (no events) --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-07-14T08:21:43Z) (no matching error/webhook lines in operator logs for this time window) --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-568d65fbfc-vklq9.log ==================== ==================== OPERATOR POD LOG: rhods-operator-568d65fbfc-zxp47.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-765c754cf5-8fz4t.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/data-science-pipelines-operator-controller-manager-6cc8f7bhk8mn.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/feast-operator-controller-manager-7b7cd769f7-42gxb.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kubeflow-trainer-controller-manager-cb88d97c9-szgbg.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/kuberay-operator-66fb75559c-nmf45.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/mlflow-operator-controller-manager-b5455d498-k8dj2.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-registry-operator-controller-manager-79765dbdfb-2zkbs.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/model-serving-api-5679d6b4f-gv92m.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-model-controller-65fdb9c5c-8nzrs.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/odh-notebook-controller-manager-598c78b776-hg8fl.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/rhods-dashboard-676945747-f6f9p.log ==================== ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-6dcf85dd5d-ld442.log ====================