=== RUN TestOdhOperator/components/group_1/sparkoperator/Validate_ScheduledSparkApplication_workload_execution helper_test.go:150: Creating ScheduledSparkApplication scheduled-spark-pi-d9em1n41feec7bbembog in namespace redhat-ods-applications helper_test.go:150: Waiting for ScheduledSparkApplication scheduled-spark-pi-d9em1n41feec7bbembog to schedule a run helper_test.go:150: ScheduledSparkApplication scheduled-spark-pi-d9em1n41feec7bbembog created SparkApplication: scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380 helper_test.go:150: Suspending ScheduledSparkApplication scheduled-spark-pi-d9em1n41feec7bbembog to prevent additional runs helper_test.go:150: Waiting for SparkApplication scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380 to reach terminal state sparkoperator_test.go:112: Cleaning up ScheduledSparkApplication scheduled-spark-pi-d9em1n41feec7bbembog sparkoperator_test.go:120: Verifying cascade deletion of SparkApplication scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380 testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/sparkoperator/Validate_ScheduledSparkApplication_workload_execution (340.14s) === RUN TestOdhOperator/components/group_1/sparkoperator === PAUSE TestOdhOperator/components/group_1/sparkoperator === CONT TestOdhOperator/components/group_1/sparkoperator 2026-07-19T23:40:19Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.25"}} test_context_test.go:223: Timed out after 300.009s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: SparkApplication scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380 (created by ScheduledSparkApplication) did not reach terminal state Expected <string>: map[apiVersion:sparkoperator.k8s.io/v1beta2 kind:SparkApplication metadata:map[creationTimestamp:2026-07-19T23:42:50Z generation:1 labels:map[sparkoperator.k8s.io/scheduled-app-name:scheduled-spark-pi-d9em1n41feec7bbembog] managedFields:[map[apiVersion:sparkoperator.k8s.io/v1beta2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:labels:map[.:map[] f:sparkoperator.k8s.io/scheduled-app-name:map[]] f:ownerReferences:map[.:map[] k:{"uid":"0320bc40-dfaf-419e-bd7d-9051e677e19c"}:map[]]] f:spec:map[.:map[] f:arguments:map[] f:deps:map[] f:driver:map[.:map[] f:coreLimit:map[] f:cores:map[] f:labels:map[.:map[] f:version:map[]] f:memory:map[] f:securityContext:map[] f:serviceAccount:map[] f:volumeMounts:map[]] f:executor:map[.:map[] f:cores:map[] f:instances:map[] f:labels:map[.:map[] f:version:map[]] f:memory:map[] f:securityContext:map[] f:volumeMounts:map[]] f:image:map[] f:imagePullPolicy:map[] f:mainApplicationFile:map[] f:mainClass:map[] f:mode:map[] f:restartPolicy:map[.:map[] f:type:map[]] f:sparkConf:map[.:map[] f:spark.blockManager.port:map[] f:spark.driver.blockManager.port:map[] f:spark.driver.port:map[] f:spark.port.maxRetries:map[]] f:sparkVersion:map[] f:type:map[] f:volumes:map[]]] manager:spark-operator operation:Update time:2026-07-19T23:42:50Z] map[apiVersion:sparkoperator.k8s.io/v1beta2 fieldsType:FieldsV1 fieldsV1:map[f:status:map[.:map[] f:applicationState:map[.:map[] f:state:map[]] f:driverInfo:map[.:map[] f:podName:map[] f:webUIAddress:map[] f:webUIPort:map[] f:webUIServiceName:map[]] f:executionAttempts:map[] f:executorState:map[.:map[] f:spark-pi-ddfeee9f7cc322c0-exec-1:map[]] f:lastSubmissionAttemptTime:map[] f:sparkApplicationId:map[] f:submissionAttempts:map[] f:submissionID:map[] f:terminationTime:map[]]] manager:spark-operator operation:Update subresource:status time:2026-07-19T23:43:15Z]] name:scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380 namespace:redhat-ods-applications ownerReferences:[map[apiVersion:sparkoperator.k8s.io/v1beta2 blockOwnerDeletion:true kind:ScheduledSparkApplication name:scheduled-spark-pi-d9em1n41feec7bbembog uid:0320bc40-dfaf-419e-bd7d-9051e677e19c]] resourceVersion:155622 uid:19a2de78-d4b4-4b2f-a011-50c7756d5531] spec:map[arguments:[1000] deps:map[] driver:map[coreLimit:1200m cores:1 labels:map[version:4.0.1] memory:512m securityContext:map[] serviceAccount:spark-operator-spark volumeMounts:[map[mountPath:/opt/spark/work-dir name:spark-work-dir]]] executor:map[cores:1 instances:1 labels:map[version:4.0.1] memory:512m securityContext:map[] volumeMounts:[map[mountPath:/opt/spark/work-dir name:spark-work-dir]]] image:quay.io/opendatahub/data-processing:Spark-v4.0.1 imagePullPolicy:IfNotPresent mainApplicationFile:local:///opt/spark/examples/jars/spark-examples_2.13-4.0.1.jar mainClass:org.apache.spark.examples.SparkPi mode:cluster restartPolicy:map[type:Never] sparkConf:map[spark.blockManager.port:8081 spark.driver.blockManager.port:8082 spark.driver.port:8080 spark.port.maxRetries:0] sparkVersion:4.0.1 type:Scala volumes:[map[emptyDir:map[] name:spark-work-dir]]] status:map[applicationState:map[state:RUNNING] driverInfo:map[podName:scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver webUIAddress:172.31.154.34:4040 webUIPort:4040 webUIServiceName:scheduled-spark-pi-d9em1n41feec7bbembog-1784504-20927d5c-ui-svc] executionAttempts:1 executorState:map[spark-pi-ddfeee9f7cc322c0-exec-1:RUNNING] lastSubmissionAttemptTime:2026-07-19T23:42:50Z sparkApplicationId:spark-c2cab82fcf0b40378275e4bd78365729 submissionAttempts:1 submissionID:779c3b8b-0328-4e8d-9317-76aab33b1df4 terminationTime:<nil>]] to match expression <string>: .status.applicationState.state == "COMPLETED" or .status.applicationState.state == "FAILED" --- FAIL: TestOdhOperator/components/group_1/sparkoperator (460.55s) === RUN TestOdhOperator/components/group_1 controller_test.go:753: Skipping tests for components/aigateway controller_test.go:753: Skipping tests for components/dashboard controller_test.go:753: Skipping tests for components/datasciencepipelines controller_test.go:753: Skipping tests for components/feastoperator controller_test.go:753: Skipping tests for components/kserve controller_test.go:753: Skipping tests for components/modelregistry controller_test.go:753: Skipping tests for components/ogx controller_test.go:753: Skipping tests for components/ray controller_test.go:753: Skipping tests for components/trainer controller_test.go:753: Skipping tests for components/trainingoperator controller_test.go:753: Skipping tests for components/workbenches --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components 2026/07/19 23:48:00 TEST FAILURE DETECTED: group 1 2026/07/19 23:48:00 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/19 23:48:00 === CLUSTER STATE === 2026/07/19 23:48:00 No nodes found in cluster 2026/07/19 23:48:00 === DEPLOYMENTS === 2026/07/19 23:48:00 Failed to collect deployment data: deployments not ready: redhat-ods-applications/dashboard-redirect (0/2), redhat-ods-applications/data-science-pipelines-operator-controller-manager (0/1), redhat-ods-applications/feast-operator-controller-manager (0/1), redhat-ods-applications/kserve-controller-manager (0/1), redhat-ods-applications/kubeflow-trainer-controller-manager (0/1), redhat-ods-applications/kubeflow-training-operator (0/1), redhat-ods-applications/kuberay-operator (0/1), redhat-ods-applications/llmisvc-controller-manager (0/1), redhat-ods-applications/maas-postgres (0/1), redhat-ods-applications/model-registry-operator-controller-manager (0/1), redhat-ods-applications/model-serving-api (0/1), redhat-ods-applications/notebook-controller-deployment (0/1), redhat-ods-applications/odh-model-controller (0/1), redhat-ods-applications/odh-notebook-controller-manager (0/1), redhat-ods-applications/rhods-dashboard (0/2), redhat-ods-applications/spark-operator-controller (0/2), redhat-ods-applications/spark-operator-webhook (0/2), redhat-ods-applications/trustyai-service-operator-controller-manager (0/1) 2026/07/19 23:48:00 === PODS === 2026/07/19 23:48:00 Failed to collect pod data: redhat-ods-applications/dashboard-redirect-66fdc9975-gkbk9: phase=Pending; redhat-ods-applications/feast-operator-controller-manager-7b7cd769f7-v7nrb: phase=Pending; redhat-ods-applications/kserve-controller-manager-dd9f6cf9-99t75: phase=Pending; redhat-ods-applications/kubeflow-training-operator-54786944d6-pcck4: phase=Pending; redhat-ods-applications/kuberay-operator-6586cdc875-sbzd7: phase=Pending; redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-4tk56: phase=Pending; redhat-ods-applications/notebook-controller-deployment-647b7d4d45-kd2vm: phase=Pending; redhat-ods-applications/rhods-dashboard-845c79958d-lbmvv: phase=Pending; redhat-ods-applications/spark-operator-controller-5dd7d48f87-72mdx: phase=Pending; redhat-ods-applications/spark-operator-controller-5dd7d48f87-8p9fc: phase=Pending; redhat-ods-applications/spark-operator-webhook-5f8768dbfd-cvpn7: phase=Pending; redhat-ods-applications/spark-operator-webhook-5f8768dbfd-czmwt: phase=Pending 2026/07/19 23:48:00 === OPERATOR STATUS === 2026/07/19 23:48:00 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-6c684c8b65-tlxhd: Pending; operator pod rhods-operator-6c684c8b65-xcgwr: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-xrgdw: Pending; dependent jobset: deployment 0/1 ready; dependent leader-worker-set: deployment 0/1 ready; pod openshift-lws-operator-697fbb9648-vt9lz: Pending; dependent cert-manager: deployment 0/1 ready; pod cert-manager-operator-controller-manager-55c444bf86-w4s7n: Pending; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-5rkhc: Pending 2026/07/19 23:48:00 === DSCI STATUS === 2026/07/19 23:48:00 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/19 23:48:00 === DSC STATUS === 2026/07/19 23:48:00 DSC default-dsc: 2026/07/19 23:48:00 Ready: True - 2026/07/19 23:48:00 ProvisioningSucceeded: True - 2026/07/19 23:48:00 ComponentsReady: True - 2026/07/19 23:48:00 AIPipelinesReady: True - 2026/07/19 23:48:00 DashboardReady: True - 2026/07/19 23:48:00 FeastOperatorReady: True - 2026/07/19 23:48:00 KserveLLMInferenceServiceDependencies: True - 2026/07/19 23:48:00 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/19 23:48:00 KserveReady: True - 2026/07/19 23:48:00 KueueReady: False - Component ManagementState is set to Removed 2026/07/19 23:48:00 LLM-D-WVADependencies: True - 2026/07/19 23:48:00 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/19 23:48:00 ModelControllerReady: True - 2026/07/19 23:48:00 ModelRegistryReady: True - 2026/07/19 23:48:00 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/19 23:48:00 OGXReady: False - Component ManagementState is set to Removed 2026/07/19 23:48:00 RayReady: True - 2026/07/19 23:48:00 SparkOperatorReady: True - 2026/07/19 23:48:00 TrainerReady: True - 2026/07/19 23:48:00 TrainingOperatorReady: True - 2026/07/19 23:48:00 TrustyAIReady: True - 2026/07/19 23:48:00 WorkbenchesReady: True - 2026/07/19 23:48:00 === RECENT EVENTS === 2026/07/19 23:48:00 23:47:59 WARN ReplicaSet/odh-notebook-controller-manager-6dd687b79: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:55 WARN ReplicaSet/rhods-dashboard-845c79958d: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:53 WARN ReplicaSet/odh-model-controller-65fdb9c5c: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:52 WARN ReplicaSet/dashboard-redirect-66fdc9975: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:51 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:51 WARN ReplicaSet/maas-postgres-bb9d55498: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:51 WARN ReplicaSet/model-registry-operator-controller-manager-64d8b9b466: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:50 WARN ReplicaSet/data-science-pipelines-operator-controller-manager-7bf7988f59: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:50 WARN ReplicaSet/kubeflow-trainer-controller-manager-b5fd84b9c: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:50 WARN ReplicaSet/trustyai-service-operator-controller-manager-5dccbd95db: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:46:07 WARN Pod/notebook-controller-deployment-647b7d4d45-kd2vm: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:07 WARN Pod/feast-operator-controller-manager-7b7cd769f7-v7nrb: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:07 WARN Pod/llmisvc-controller-manager-7b46b9fbbb-4tk56: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:07 WARN Pod/kserve-controller-manager-dd9f6cf9-99t75: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:07 INFO ReplicaSet/notebook-controller-deployment-647b7d4d45: SuccessfulCreate - Created pod: notebook-controller-deployment-647b7d4d45-kd2vm 2026/07/19 23:48:00 23:46:07 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-99t75 2026/07/19 23:48:00 23:46:07 INFO ReplicaSet/feast-operator-controller-manager-7b7cd769f7: SuccessfulCreate - Created pod: feast-operator-controller-manager-7b7cd769f7-v7nrb 2026/07/19 23:48:00 23:46:07 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-4tk56 2026/07/19 23:48:00 23:46:06 WARN Pod/spark-operator-controller-5dd7d48f87-8p9fc: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:06 WARN Pod/rhods-dashboard-845c79958d-lbmvv: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:06 WARN Pod/dashboard-redirect-66fdc9975-gkbk9: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:06 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-8p9fc 2026/07/19 23:48:00 23:46:06 INFO ReplicaSet/dashboard-redirect-66fdc9975: SuccessfulCreate - Created pod: dashboard-redirect-66fdc9975-gkbk9 2026/07/19 23:48:00 23:46:06 INFO ReplicaSet/rhods-dashboard-845c79958d: SuccessfulCreate - Created pod: rhods-dashboard-845c79958d-lbmvv 2026/07/19 23:48:00 23:45:06 WARN Pod/kubeflow-training-operator-54786944d6-pcck4: FailedScheduling - 0/2 nodes are available: 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:45:06 INFO ReplicaSet/kubeflow-training-operator-54786944d6: SuccessfulCreate - Created pod: kubeflow-training-operator-54786944d6-pcck4 2026/07/19 23:48:00 23:45:05 WARN Pod/spark-operator-controller-5dd7d48f87-72mdx: FailedScheduling - 0/2 nodes are available: 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:45:05 WARN Pod/kuberay-operator-6586cdc875-sbzd7: FailedScheduling - 0/2 nodes are available: 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:45:05 WARN Pod/spark-operator-webhook-5f8768dbfd-czmwt: FailedScheduling - 0/2 nodes are available: 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:45:05 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-72mdx 2026/07/19 23:48:00 23:45:05 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-czmwt 2026/07/19 23:48:00 23:45:05 INFO ReplicaSet/kuberay-operator-6586cdc875: SuccessfulCreate - Created pod: kuberay-operator-6586cdc875-sbzd7 2026/07/19 23:48:00 23:45:04 WARN Pod/spark-operator-webhook-5f8768dbfd-cvpn7: FailedScheduling - 0/2 nodes are available: 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:45:04 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-cvpn7 2026/07/19 23:48:00 23:45:04 WARN Service/spark-operator-webhook-svc: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/spark-operator-webhook-svc: skipping Pod spark-operator-webhook-5f8768dbfd-h7l8c for Service redhat-ods-applications/spark-operator-webhook-svc: Node ip-10-0-128-128.ec2.internal Not Found 2026/07/19 23:48:00 23:44:12 INFO Pod/spark-operator-webhook-5f8768dbfd-dzl6m: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-dzl6m 2026/07/19 23:48:00 23:44:12 INFO Pod/spark-operator-controller-5dd7d48f87-sf7gw: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-sf7gw 2026/07/19 23:48:00 23:44:12 WARN Pod/odh-notebook-controller-manager-6dd687b79-bgg5r: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 WARN Pod/notebook-controller-deployment-647b7d4d45-bkcdd: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 INFO Pod/spark-operator-webhook-5f8768dbfd-h7l8c: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-h7l8c 2026/07/19 23:48:00 23:44:12 WARN Pod/kserve-controller-manager-dd9f6cf9-sj4k4: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 WARN Pod/rhods-dashboard-845c79958d-tqzch: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 WARN Pod/model-registry-operator-controller-manager-64d8b9b466-5qhvc: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 WARN Pod/data-science-pipelines-operator-controller-manager-7bf7988j6ps9: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 INFO Pod/kubeflow-training-operator-54786944d6-9jrkh: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/kubeflow-training-operator-54786944d6-9jrkh 2026/07/19 23:48:00 23:44:12 WARN Pod/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 INFO Pod/spark-pi-ddfeee9f7cc322c0-exec-1: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-pi-ddfeee9f7cc322c0-exec-1 2026/07/19 23:48:00 23:44:12 INFO Pod/kuberay-operator-6586cdc875-7bk9w: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/kuberay-operator-6586cdc875-7bk9w 2026/07/19 23:48:00 23:44:12 WARN Pod/dashboard-redirect-66fdc9975-qfmbn: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 WARN Pod/global-pull-secret-syncer-7xnlk: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 INFO Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver 2026/07/19 23:48:00 23:44:11 WARN Pod/dashboard-redirect-66fdc9975-vcgl6: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/spark-operator-controller-5dd7d48f87-w6h46: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/konnectivity-agent-7w82g: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/feast-operator-controller-manager-7b7cd769f7-m2flq: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/konnectivity-agent-dv7hd: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/trustyai-service-operator-controller-manager-5dccbd95db-w7t2c: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/llmisvc-controller-manager-7b46b9fbbb-n6992: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/model-serving-api-5679d6b4f-kzs5z: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/global-pull-secret-syncer-hlb82: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/rhods-dashboard-845c79958d-r28gw: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/odh-model-controller-65fdb9c5c-pg47v: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/maas-postgres-bb9d55498-8qwwc: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/kube-apiserver-proxy-ip-10-0-139-21.ec2.internal: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/kubeflow-trainer-controller-manager-b5fd84b9c-xlx2j: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:06 WARN Pod/kubeflow-training-operator-54786944d6-9jrkh: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:06 WARN Pod/kuberay-operator-6586cdc875-7bk9w: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:06 WARN Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:06 WARN Pod/spark-operator-webhook-5f8768dbfd-h7l8c: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:06 WARN Pod/konnectivity-agent-kfg5p: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:05 WARN Pod/spark-operator-webhook-5f8768dbfd-dzl6m: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:05 WARN Pod/spark-pi-ddfeee9f7cc322c0-exec-1: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:05 WARN Pod/spark-operator-controller-5dd7d48f87-sf7gw: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:05 WARN Pod/global-pull-secret-syncer-4h494: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:05 WARN Pod/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:43:15 INFO SparkApplication/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380: SparkExecutorRunning - Executor spark-pi-ddfeee9f7cc322c0-exec-1 is running 2026/07/19 23:48:00 23:43:14 INFO Pod/spark-pi-ddfeee9f7cc322c0-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-ddfeee9f7cc322c0-exec-1 to ip-10-0-128-128.ec2.internal 2026/07/19 23:48:00 23:43:14 INFO Pod/spark-pi-ddfeee9f7cc322c0-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/19 23:48:00 23:43:14 INFO Pod/spark-pi-ddfeee9f7cc322c0-exec-1: Started - Started container spark-kubernetes-executor 2026/07/19 23:48:00 23:43:14 INFO Pod/spark-pi-ddfeee9f7cc322c0-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/19 23:48:00 23:43:14 INFO Pod/spark-pi-ddfeee9f7cc322c0-exec-1: AddedInterface - Add eth0 [10.134.0.148/23] from ovn-kubernetes 2026/07/19 23:48:00 23:43:14 INFO SparkApplication/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380: SparkExecutorPending - Executor spark-pi-ddfeee9f7cc322c0-exec-1 is pending 2026/07/19 23:48:00 23:43:06 INFO SparkApplication/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380: SparkDriverRunning - Driver scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver is running 2026/07/19 23:48:00 23:43:05 INFO Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: Started - Started container spark-kubernetes-driver 2026/07/19 23:48:00 23:43:05 INFO Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: Created - Created container: spark-kubernetes-driver 2026/07/19 23:48:00 23:43:05 INFO Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/19 23:48:00 23:43:05 INFO Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: AddedInterface - Add eth0 [10.134.0.147/23] from ovn-kubernetes 2026/07/19 23:48:00 23:43:04 INFO SparkApplication/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380: SparkApplicationSubmitted - SparkApplication scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380 was submitted successfully 2026/07/19 23:48:00 23:43:04 WARN Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-f93c6f9f7cc2eea9-driver-podspec-conf-map" not found 2026/07/19 23:48:00 23:43:04 WARN Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-578c489f7cc300b2-conf-map" not found 2026/07/19 23:48:00 23:43:03 INFO Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: Scheduled - Successfully assigned redhat-ods-applications/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver to ip-10-0-128-128.ec2.internal 2026/07/19 23:48:00 === RESOURCE QUOTAS === 2026/07/19 23:48:00 No resource quotas found in monitored namespaces 2026/07/19 23:48:00 Classification for group 1: infrastructure/pod-startup (code=1002, confidence=high) 2026/07/19 23:48:00 Evidence: pod dashboard-redirect-66fdc9975-gkbk9 stuck in Pending for 1m54s FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["pod dashboard-redirect-66fdc9975-gkbk9 stuck in Pending for 1m54s"],"confidence":"high"} 2026/07/19 23:48:00 Diagnostics complete for failed test: group 1 controller_test.go:753: Stopping: group 1 test failed. 2026/07/19 23:48:00 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/19 23:48:00 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/19 23:48:00 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (460.66s) === RUN TestOdhOperator 2026/07/19 23:40:09 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/19 23:48:00 TEST FAILURE DETECTED: components 2026/07/19 23:48:00 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/19 23:48:00 === CLUSTER STATE === 2026/07/19 23:48:00 No nodes found in cluster 2026/07/19 23:48:00 === DEPLOYMENTS === 2026/07/19 23:48:00 Failed to collect deployment data: deployments not ready: redhat-ods-applications/dashboard-redirect (0/2), redhat-ods-applications/data-science-pipelines-operator-controller-manager (0/1), redhat-ods-applications/feast-operator-controller-manager (0/1), redhat-ods-applications/kserve-controller-manager (0/1), redhat-ods-applications/kubeflow-trainer-controller-manager (0/1), redhat-ods-applications/kubeflow-training-operator (0/1), redhat-ods-applications/kuberay-operator (0/1), redhat-ods-applications/llmisvc-controller-manager (0/1), redhat-ods-applications/maas-postgres (0/1), redhat-ods-applications/model-registry-operator-controller-manager (0/1), redhat-ods-applications/model-serving-api (0/1), redhat-ods-applications/notebook-controller-deployment (0/1), redhat-ods-applications/odh-model-controller (0/1), redhat-ods-applications/odh-notebook-controller-manager (0/1), redhat-ods-applications/rhods-dashboard (0/2), redhat-ods-applications/spark-operator-controller (0/2), redhat-ods-applications/spark-operator-webhook (0/2), redhat-ods-applications/trustyai-service-operator-controller-manager (0/1) 2026/07/19 23:48:00 === PODS === 2026/07/19 23:48:00 Failed to collect pod data: redhat-ods-applications/dashboard-redirect-66fdc9975-gkbk9: phase=Pending; redhat-ods-applications/feast-operator-controller-manager-7b7cd769f7-v7nrb: phase=Pending; redhat-ods-applications/kserve-controller-manager-dd9f6cf9-99t75: phase=Pending; redhat-ods-applications/kubeflow-training-operator-54786944d6-pcck4: phase=Pending; redhat-ods-applications/kuberay-operator-6586cdc875-sbzd7: phase=Pending; redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-4tk56: phase=Pending; redhat-ods-applications/notebook-controller-deployment-647b7d4d45-kd2vm: phase=Pending; redhat-ods-applications/rhods-dashboard-845c79958d-lbmvv: phase=Pending; redhat-ods-applications/spark-operator-controller-5dd7d48f87-72mdx: phase=Pending; redhat-ods-applications/spark-operator-controller-5dd7d48f87-8p9fc: phase=Pending; redhat-ods-applications/spark-operator-webhook-5f8768dbfd-cvpn7: phase=Pending; redhat-ods-applications/spark-operator-webhook-5f8768dbfd-czmwt: phase=Pending 2026/07/19 23:48:00 === OPERATOR STATUS === 2026/07/19 23:48:00 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-6c684c8b65-tlxhd: Pending; operator pod rhods-operator-6c684c8b65-xcgwr: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-xrgdw: Pending; dependent jobset: deployment 0/1 ready; dependent leader-worker-set: deployment 0/1 ready; pod openshift-lws-operator-697fbb9648-vt9lz: Pending; dependent cert-manager: deployment 0/1 ready; pod cert-manager-operator-controller-manager-55c444bf86-w4s7n: Pending; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-5rkhc: Pending 2026/07/19 23:48:00 === DSCI STATUS === 2026/07/19 23:48:00 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/19 23:48:00 === DSC STATUS === 2026/07/19 23:48:00 DSC default-dsc: 2026/07/19 23:48:00 Ready: True - 2026/07/19 23:48:00 ProvisioningSucceeded: True - 2026/07/19 23:48:00 ComponentsReady: True - 2026/07/19 23:48:00 AIPipelinesReady: True - 2026/07/19 23:48:00 DashboardReady: True - 2026/07/19 23:48:00 FeastOperatorReady: True - 2026/07/19 23:48:00 KserveLLMInferenceServiceDependencies: True - 2026/07/19 23:48:00 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/19 23:48:00 KserveReady: True - 2026/07/19 23:48:00 KueueReady: False - Component ManagementState is set to Removed 2026/07/19 23:48:00 LLM-D-WVADependencies: True - 2026/07/19 23:48:00 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/19 23:48:00 ModelControllerReady: True - 2026/07/19 23:48:00 ModelRegistryReady: True - 2026/07/19 23:48:00 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/19 23:48:00 OGXReady: False - Component ManagementState is set to Removed 2026/07/19 23:48:00 RayReady: True - 2026/07/19 23:48:00 SparkOperatorReady: True - 2026/07/19 23:48:00 TrainerReady: True - 2026/07/19 23:48:00 TrainingOperatorReady: True - 2026/07/19 23:48:00 TrustyAIReady: True - 2026/07/19 23:48:00 WorkbenchesReady: True - 2026/07/19 23:48:00 === RECENT EVENTS === 2026/07/19 23:48:00 23:47:59 WARN ReplicaSet/odh-notebook-controller-manager-6dd687b79: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:55 WARN ReplicaSet/rhods-dashboard-845c79958d: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:53 WARN ReplicaSet/odh-model-controller-65fdb9c5c: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:52 WARN ReplicaSet/dashboard-redirect-66fdc9975: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:51 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:51 WARN ReplicaSet/maas-postgres-bb9d55498: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:51 WARN ReplicaSet/model-registry-operator-controller-manager-64d8b9b466: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:50 WARN ReplicaSet/data-science-pipelines-operator-controller-manager-7bf7988f59: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:50 WARN ReplicaSet/kubeflow-trainer-controller-manager-b5fd84b9c: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:47:50 WARN ReplicaSet/trustyai-service-operator-controller-manager-5dccbd95db: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/19 23:48:00 23:46:07 WARN Pod/notebook-controller-deployment-647b7d4d45-kd2vm: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:07 WARN Pod/feast-operator-controller-manager-7b7cd769f7-v7nrb: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:07 WARN Pod/llmisvc-controller-manager-7b46b9fbbb-4tk56: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:07 WARN Pod/kserve-controller-manager-dd9f6cf9-99t75: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:07 INFO ReplicaSet/notebook-controller-deployment-647b7d4d45: SuccessfulCreate - Created pod: notebook-controller-deployment-647b7d4d45-kd2vm 2026/07/19 23:48:00 23:46:07 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-99t75 2026/07/19 23:48:00 23:46:07 INFO ReplicaSet/feast-operator-controller-manager-7b7cd769f7: SuccessfulCreate - Created pod: feast-operator-controller-manager-7b7cd769f7-v7nrb 2026/07/19 23:48:00 23:46:07 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-4tk56 2026/07/19 23:48:00 23:46:06 WARN Pod/spark-operator-controller-5dd7d48f87-8p9fc: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:06 WARN Pod/rhods-dashboard-845c79958d-lbmvv: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:06 WARN Pod/dashboard-redirect-66fdc9975-gkbk9: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:46:06 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-8p9fc 2026/07/19 23:48:00 23:46:06 INFO ReplicaSet/dashboard-redirect-66fdc9975: SuccessfulCreate - Created pod: dashboard-redirect-66fdc9975-gkbk9 2026/07/19 23:48:00 23:46:06 INFO ReplicaSet/rhods-dashboard-845c79958d: SuccessfulCreate - Created pod: rhods-dashboard-845c79958d-lbmvv 2026/07/19 23:48:00 23:45:06 WARN Pod/kubeflow-training-operator-54786944d6-pcck4: FailedScheduling - 0/2 nodes are available: 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:45:06 INFO ReplicaSet/kubeflow-training-operator-54786944d6: SuccessfulCreate - Created pod: kubeflow-training-operator-54786944d6-pcck4 2026/07/19 23:48:00 23:45:05 WARN Pod/spark-operator-controller-5dd7d48f87-72mdx: FailedScheduling - 0/2 nodes are available: 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:45:05 WARN Pod/kuberay-operator-6586cdc875-sbzd7: FailedScheduling - 0/2 nodes are available: 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:45:05 WARN Pod/spark-operator-webhook-5f8768dbfd-czmwt: FailedScheduling - 0/2 nodes are available: 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:45:05 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-72mdx 2026/07/19 23:48:00 23:45:05 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-czmwt 2026/07/19 23:48:00 23:45:05 INFO ReplicaSet/kuberay-operator-6586cdc875: SuccessfulCreate - Created pod: kuberay-operator-6586cdc875-sbzd7 2026/07/19 23:48:00 23:45:04 WARN Pod/spark-operator-webhook-5f8768dbfd-cvpn7: FailedScheduling - 0/2 nodes are available: 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. 2026/07/19 23:48:00 23:45:04 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-cvpn7 2026/07/19 23:48:00 23:45:04 WARN Service/spark-operator-webhook-svc: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/spark-operator-webhook-svc: skipping Pod spark-operator-webhook-5f8768dbfd-h7l8c for Service redhat-ods-applications/spark-operator-webhook-svc: Node ip-10-0-128-128.ec2.internal Not Found 2026/07/19 23:48:00 23:44:12 INFO Pod/spark-operator-webhook-5f8768dbfd-dzl6m: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-dzl6m 2026/07/19 23:48:00 23:44:12 INFO Pod/spark-operator-controller-5dd7d48f87-sf7gw: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-sf7gw 2026/07/19 23:48:00 23:44:12 WARN Pod/odh-notebook-controller-manager-6dd687b79-bgg5r: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 WARN Pod/notebook-controller-deployment-647b7d4d45-bkcdd: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 INFO Pod/spark-operator-webhook-5f8768dbfd-h7l8c: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-h7l8c 2026/07/19 23:48:00 23:44:12 WARN Pod/kserve-controller-manager-dd9f6cf9-sj4k4: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 WARN Pod/rhods-dashboard-845c79958d-tqzch: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 WARN Pod/model-registry-operator-controller-manager-64d8b9b466-5qhvc: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 WARN Pod/data-science-pipelines-operator-controller-manager-7bf7988j6ps9: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 INFO Pod/kubeflow-training-operator-54786944d6-9jrkh: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/kubeflow-training-operator-54786944d6-9jrkh 2026/07/19 23:48:00 23:44:12 WARN Pod/kube-apiserver-proxy-ip-10-0-133-201.ec2.internal: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 INFO Pod/spark-pi-ddfeee9f7cc322c0-exec-1: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-pi-ddfeee9f7cc322c0-exec-1 2026/07/19 23:48:00 23:44:12 INFO Pod/kuberay-operator-6586cdc875-7bk9w: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/kuberay-operator-6586cdc875-7bk9w 2026/07/19 23:48:00 23:44:12 WARN Pod/dashboard-redirect-66fdc9975-qfmbn: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 WARN Pod/global-pull-secret-syncer-7xnlk: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:12 INFO Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver 2026/07/19 23:48:00 23:44:11 WARN Pod/dashboard-redirect-66fdc9975-vcgl6: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/spark-operator-controller-5dd7d48f87-w6h46: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/konnectivity-agent-7w82g: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/feast-operator-controller-manager-7b7cd769f7-m2flq: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/konnectivity-agent-dv7hd: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/trustyai-service-operator-controller-manager-5dccbd95db-w7t2c: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/llmisvc-controller-manager-7b46b9fbbb-n6992: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/model-serving-api-5679d6b4f-kzs5z: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/global-pull-secret-syncer-hlb82: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/rhods-dashboard-845c79958d-r28gw: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/odh-model-controller-65fdb9c5c-pg47v: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/maas-postgres-bb9d55498-8qwwc: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/kube-apiserver-proxy-ip-10-0-139-21.ec2.internal: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:11 WARN Pod/kubeflow-trainer-controller-manager-b5fd84b9c-xlx2j: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:06 WARN Pod/kubeflow-training-operator-54786944d6-9jrkh: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:06 WARN Pod/kuberay-operator-6586cdc875-7bk9w: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:06 WARN Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:06 WARN Pod/spark-operator-webhook-5f8768dbfd-h7l8c: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:06 WARN Pod/konnectivity-agent-kfg5p: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:05 WARN Pod/spark-operator-webhook-5f8768dbfd-dzl6m: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:05 WARN Pod/spark-pi-ddfeee9f7cc322c0-exec-1: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:05 WARN Pod/spark-operator-controller-5dd7d48f87-sf7gw: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:05 WARN Pod/global-pull-secret-syncer-4h494: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:44:05 WARN Pod/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal: NodeNotReady - Node is not ready 2026/07/19 23:48:00 23:43:15 INFO SparkApplication/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380: SparkExecutorRunning - Executor spark-pi-ddfeee9f7cc322c0-exec-1 is running 2026/07/19 23:48:00 23:43:14 INFO Pod/spark-pi-ddfeee9f7cc322c0-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-ddfeee9f7cc322c0-exec-1 to ip-10-0-128-128.ec2.internal 2026/07/19 23:48:00 23:43:14 INFO Pod/spark-pi-ddfeee9f7cc322c0-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/19 23:48:00 23:43:14 INFO Pod/spark-pi-ddfeee9f7cc322c0-exec-1: Started - Started container spark-kubernetes-executor 2026/07/19 23:48:00 23:43:14 INFO Pod/spark-pi-ddfeee9f7cc322c0-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/19 23:48:00 23:43:14 INFO Pod/spark-pi-ddfeee9f7cc322c0-exec-1: AddedInterface - Add eth0 [10.134.0.148/23] from ovn-kubernetes 2026/07/19 23:48:00 23:43:14 INFO SparkApplication/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380: SparkExecutorPending - Executor spark-pi-ddfeee9f7cc322c0-exec-1 is pending 2026/07/19 23:48:00 23:43:06 INFO SparkApplication/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380: SparkDriverRunning - Driver scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver is running 2026/07/19 23:48:00 23:43:05 INFO Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: Started - Started container spark-kubernetes-driver 2026/07/19 23:48:00 23:43:05 INFO Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: Created - Created container: spark-kubernetes-driver 2026/07/19 23:48:00 23:43:05 INFO Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/19 23:48:00 23:43:05 INFO Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: AddedInterface - Add eth0 [10.134.0.147/23] from ovn-kubernetes 2026/07/19 23:48:00 23:43:04 INFO SparkApplication/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380: SparkApplicationSubmitted - SparkApplication scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380 was submitted successfully 2026/07/19 23:48:00 23:43:04 WARN Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-f93c6f9f7cc2eea9-driver-podspec-conf-map" not found 2026/07/19 23:48:00 23:43:04 WARN Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-578c489f7cc300b2-conf-map" not found 2026/07/19 23:48:00 23:43:03 INFO Pod/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver: Scheduled - Successfully assigned redhat-ods-applications/scheduled-spark-pi-d9em1n41feec7bbembog-1784504570000862380-driver to ip-10-0-128-128.ec2.internal 2026/07/19 23:48:00 === RESOURCE QUOTAS === 2026/07/19 23:48:00 No resource quotas found in monitored namespaces 2026/07/19 23:48:00 Classification for components: infrastructure/pod-startup (code=1002, confidence=high) 2026/07/19 23:48:00 Evidence: pod dashboard-redirect-66fdc9975-gkbk9 stuck in Pending for 1m54s FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["pod dashboard-redirect-66fdc9975-gkbk9 stuck in Pending for 1m54s"],"confidence":"high"} 2026/07/19 23:48:00 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/19 23:48:00 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/19 23:48:00 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/19 23:48:00 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (471.40s)