=== RUN TestOdhOperator/components/group_1/sparkoperator/Validate_SparkPi_workload_execution helper_test.go:150: Creating SparkApplication spark-pi-d9ei7m74nvms7b57vg8g in namespace redhat-ods-applications helper_test.go:150: Waiting for SparkApplication spark-pi-d9ei7m74nvms7b57vg8g to complete sparkoperator_test.go:72: Cleaning up SparkApplication spark-pi-d9ei7m74nvms7b57vg8g testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/sparkoperator/Validate_SparkPi_workload_execution (300.09s) === RUN TestOdhOperator/components/group_1/sparkoperator === PAUSE TestOdhOperator/components/group_1/sparkoperator === CONT TestOdhOperator/components/group_1/sparkoperator 2026-07-19T19:20:50Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.25"}} test_context_test.go:223: Timed out after 300.009s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: SparkApplication spark-pi-d9ei7m74nvms7b57vg8g did not complete successfully Expected <string>: map[apiVersion:sparkoperator.k8s.io/v1beta2 kind:SparkApplication metadata:map[creationTimestamp:2026-07-19T19:22:00Z generation:1 managedFields:[map[apiVersion:sparkoperator.k8s.io/v1beta2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[.:map[] f:arguments:map[] f:driver:map[.:map[] f:coreLimit:map[] f:cores:map[] f:labels:map[.:map[] f:version:map[]] f:memory:map[] f:securityContext:map[] f:serviceAccount:map[] f:volumeMounts:map[]] f:executor:map[.:map[] f:cores:map[] f:instances:map[] f:labels:map[.:map[] f:version:map[]] f:memory:map[] f:securityContext:map[] f:volumeMounts:map[]] f:image:map[] f:imagePullPolicy:map[] f:mainApplicationFile:map[] f:mainClass:map[] f:mode:map[] f:restartPolicy:map[.:map[] f:type:map[]] f:sparkConf:map[.:map[] f:spark.blockManager.port:map[] f:spark.driver.blockManager.port:map[] f:spark.driver.port:map[] f:spark.port.maxRetries:map[]] f:sparkVersion:map[] f:type:map[] f:volumes:map[]]] manager:e2e-tests operation:Update time:2026-07-19T19:22:00Z] map[apiVersion:sparkoperator.k8s.io/v1beta2 fieldsType:FieldsV1 fieldsV1:map[f:status:map[.:map[] f:applicationState:map[.:map[] f:state:map[]] f:driverInfo:map[.:map[] f:podName:map[] f:webUIAddress:map[] f:webUIPort:map[] f:webUIServiceName:map[]] f:executionAttempts:map[] f:executorState:map[.:map[] f:spark-pi-d7dcb59f7bd45831-exec-1:map[]] f:lastSubmissionAttemptTime:map[] f:sparkApplicationId:map[] f:submissionAttempts:map[] f:submissionID:map[] f:terminationTime:map[]]] manager:spark-operator operation:Update subresource:status time:2026-07-19T19:22:24Z]] name:spark-pi-d9ei7m74nvms7b57vg8g namespace:redhat-ods-applications resourceVersion:155169 uid:d5b59bcb-51b5-4416-aea5-f9f76eac4a72] spec:map[arguments:[1000] deps:map[] driver:map[coreLimit:1200m cores:1 labels:map[version:4.0.1] memory:512m securityContext:map[] serviceAccount:spark-operator-spark volumeMounts:[map[mountPath:/opt/spark/work-dir name:spark-work-dir]]] executor:map[cores:1 instances:1 labels:map[version:4.0.1] memory:512m securityContext:map[] volumeMounts:[map[mountPath:/opt/spark/work-dir name:spark-work-dir]]] image:quay.io/opendatahub/data-processing:Spark-v4.0.1 imagePullPolicy:IfNotPresent mainApplicationFile:local:///opt/spark/examples/jars/spark-examples_2.13-4.0.1.jar mainClass:org.apache.spark.examples.SparkPi mode:cluster restartPolicy:map[type:Never] sparkConf:map[spark.blockManager.port:8081 spark.driver.blockManager.port:8082 spark.driver.port:8080 spark.port.maxRetries:0] sparkVersion:4.0.1 type:Scala volumes:[map[emptyDir:map[] name:spark-work-dir]]] status:map[applicationState:map[state:RUNNING] driverInfo:map[podName:spark-pi-d9ei7m74nvms7b57vg8g-driver webUIAddress:172.31.68.212:4040 webUIPort:4040 webUIServiceName:spark-pi-d9ei7m74nvms7b57vg8g-ui-svc] executionAttempts:1 executorState:map[spark-pi-d7dcb59f7bd45831-exec-1:PENDING] lastSubmissionAttemptTime:2026-07-19T19:22:00Z sparkApplicationId:spark-4ba45b37985a443da434263d31ed72d7 submissionAttempts:1 submissionID:3f1088a5-0eb3-4d0e-8e82-22cf264a4e59 terminationTime:<nil>]] to match expression <string>: .status.applicationState.state == "COMPLETED" --- FAIL: TestOdhOperator/components/group_1/sparkoperator (370.45s) === RUN TestOdhOperator/components/group_1 controller_test.go:753: Skipping tests for components/aigateway controller_test.go:753: Skipping tests for components/dashboard controller_test.go:753: Skipping tests for components/datasciencepipelines controller_test.go:753: Skipping tests for components/feastoperator controller_test.go:753: Skipping tests for components/kserve controller_test.go:753: Skipping tests for components/modelregistry controller_test.go:753: Skipping tests for components/ogx controller_test.go:753: Skipping tests for components/ray controller_test.go:753: Skipping tests for components/trainer controller_test.go:753: Skipping tests for components/trainingoperator controller_test.go:753: Skipping tests for components/workbenches --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components 2026/07/19 19:27:00 TEST FAILURE DETECTED: group 1 2026/07/19 19:27:00 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/19 19:27:00 clusterhealth: node metrics unavailable (Metrics Server may not be installed): the server is currently unable to handle the request (get nodes.metrics.k8s.io) 2026/07/19 19:27:00 === CLUSTER STATE === 2026/07/19 19:27:00 Failed to collect node data: unhealthy nodes: ip-10-0-141-80.ec2.internal (Ready=Unknown) 2026/07/19 19:27:00 === DEPLOYMENTS === 2026/07/19 19:27:00 Failed to collect deployment data: deployments not ready: redhat-ods-applications/dashboard-redirect (0/2), redhat-ods-applications/data-science-pipelines-operator-controller-manager (0/1), redhat-ods-applications/feast-operator-controller-manager (0/1), redhat-ods-applications/kserve-controller-manager (0/1), redhat-ods-applications/kubeflow-trainer-controller-manager (0/1), redhat-ods-applications/kubeflow-training-operator (0/1), redhat-ods-applications/kuberay-operator (0/1), redhat-ods-applications/llmisvc-controller-manager (0/1), redhat-ods-applications/maas-postgres (0/1), redhat-ods-applications/model-registry-operator-controller-manager (0/1), redhat-ods-applications/model-serving-api (0/1), redhat-ods-applications/notebook-controller-deployment (0/1), redhat-ods-applications/odh-model-controller (0/1), redhat-ods-applications/odh-notebook-controller-manager (0/1), redhat-ods-applications/rhods-dashboard (0/2), redhat-ods-applications/spark-operator-controller (0/2), redhat-ods-applications/spark-operator-webhook (0/2), redhat-ods-applications/trustyai-service-operator-controller-manager (0/1) 2026/07/19 19:27:00 === PODS === 2026/07/19 19:27:00 Failed to collect pod data: redhat-ods-applications/dashboard-redirect-779874bd57-hpm6n: phase=Pending; redhat-ods-applications/data-science-pipelines-operator-controller-manager-7bf7988pmrqn: phase=Pending; redhat-ods-applications/feast-operator-controller-manager-7b7cd769f7-5t994: phase=Pending; redhat-ods-applications/kserve-controller-manager-dd9f6cf9-zgfp2: phase=Pending; redhat-ods-applications/kubeflow-trainer-controller-manager-b5fd84b9c-qvkwb: phase=Pending; redhat-ods-applications/kubeflow-training-operator-54786944d6-8cz6d: phase=Pending; redhat-ods-applications/kuberay-operator-6586cdc875-gfgkd: phase=Pending; redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-sn7lz: phase=Pending; redhat-ods-applications/maas-postgres-bb9d55498-k2hcr: phase=Pending; redhat-ods-applications/model-registry-operator-controller-manager-777cd8666c-7lv7j: phase=Pending; redhat-ods-applications/model-serving-api-5679d6b4f-n77rx: phase=Pending; redhat-ods-applications/notebook-controller-deployment-647b7d4d45-5hpjg: phase=Pending; redhat-ods-applications/odh-model-controller-65fdb9c5c-4h5f5: phase=Pending; redhat-ods-applications/rhods-dashboard-6fbb47df4b-56llp: phase=Pending; redhat-ods-applications/spark-operator-controller-5dd7d48f87-2kdj4: phase=Pending; redhat-ods-applications/spark-operator-controller-5dd7d48f87-qfcc6: phase=Pending; redhat-ods-applications/spark-operator-webhook-5f8768dbfd-4x44h: phase=Pending; redhat-ods-applications/spark-operator-webhook-5f8768dbfd-cdn48: phase=Pending; redhat-ods-applications/trustyai-service-operator-controller-manager-5dccbd95db-s5jlt: phase=Pending 2026/07/19 19:27:00 === OPERATOR STATUS === 2026/07/19 19:27:00 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-5d8f6b8fb-dhdgv: Pending; operator pod rhods-operator-5d8f6b8fb-m4d8c: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-p6djc: Pending; dependent jobset: deployment 0/1 ready; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; pod cert-manager-operator-controller-manager-55c444bf86-dkb8m: Pending; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-z2vcf: Pending 2026/07/19 19:27:00 === DSCI STATUS === 2026/07/19 19:27:00 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/19 19:27:00 === DSC STATUS === 2026/07/19 19:27:00 DSC default-dsc: 2026/07/19 19:27:00 Ready: True - 2026/07/19 19:27:00 ProvisioningSucceeded: True - 2026/07/19 19:27:00 ComponentsReady: True - 2026/07/19 19:27:00 AIPipelinesReady: True - 2026/07/19 19:27:00 DashboardReady: True - 2026/07/19 19:27:00 FeastOperatorReady: True - 2026/07/19 19:27:00 KserveLLMInferenceServiceDependencies: True - 2026/07/19 19:27:00 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/19 19:27:00 KserveReady: True - 2026/07/19 19:27:00 KueueReady: False - Component ManagementState is set to Removed 2026/07/19 19:27:00 LLM-D-WVADependencies: True - 2026/07/19 19:27:00 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/19 19:27:00 ModelControllerReady: True - 2026/07/19 19:27:00 ModelRegistryReady: True - 2026/07/19 19:27:00 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/19 19:27:00 OGXReady: False - Component ManagementState is set to Removed 2026/07/19 19:27:00 RayReady: True - 2026/07/19 19:27:00 SparkOperatorReady: True - 2026/07/19 19:27:00 TrainerReady: True - 2026/07/19 19:27:00 TrainingOperatorReady: True - 2026/07/19 19:27:00 TrustyAIReady: True - 2026/07/19 19:27:00 WorkbenchesReady: True - 2026/07/19 19:27:00 === RECENT EVENTS === 2026/07/19 19:27:00 19:26:06 WARN Pod/rhods-dashboard-6fbb47df4b-56llp: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:06 WARN Pod/feast-operator-controller-manager-7b7cd769f7-5t994: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:06 WARN Pod/kubeflow-training-operator-54786944d6-8cz6d: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:06 WARN Pod/spark-operator-webhook-5f8768dbfd-4x44h: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:06 INFO ReplicaSet/feast-operator-controller-manager-7b7cd769f7: SuccessfulCreate - Created pod: feast-operator-controller-manager-7b7cd769f7-5t994 2026/07/19 19:27:00 19:26:06 INFO ReplicaSet/kubeflow-training-operator-54786944d6: SuccessfulCreate - Created pod: kubeflow-training-operator-54786944d6-8cz6d 2026/07/19 19:27:00 19:26:06 INFO ReplicaSet/rhods-dashboard-6fbb47df4b: SuccessfulCreate - Created pod: rhods-dashboard-6fbb47df4b-56llp 2026/07/19 19:27:00 19:26:05 WARN Pod/trustyai-service-operator-controller-manager-5dccbd95db-s5jlt: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:05 WARN Pod/model-serving-api-5679d6b4f-n77rx: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:05 WARN Pod/llmisvc-controller-manager-7b46b9fbbb-sn7lz: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:05 WARN Pod/spark-operator-webhook-5f8768dbfd-cdn48: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:05 WARN Pod/spark-operator-controller-5dd7d48f87-2kdj4: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:05 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-4x44h 2026/07/19 19:27:00 19:26:05 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-sn7lz 2026/07/19 19:27:00 19:26:05 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-n77rx 2026/07/19 19:27:00 19:26:05 WARN Service/spark-operator-webhook-svc: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/spark-operator-webhook-svc: skipping Pod spark-operator-webhook-5f8768dbfd-4w5s8 for Service redhat-ods-applications/spark-operator-webhook-svc: Node ip-10-0-135-209.ec2.internal Not Found 2026/07/19 19:27:00 19:26:05 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-cdn48 2026/07/19 19:27:00 19:26:05 INFO ReplicaSet/trustyai-service-operator-controller-manager-5dccbd95db: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-5dccbd95db-s5jlt 2026/07/19 19:27:00 19:26:05 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-2kdj4 2026/07/19 19:27:00 19:26:04 WARN Pod/spark-operator-controller-5dd7d48f87-qfcc6: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:04 WARN Pod/kuberay-operator-6586cdc875-gfgkd: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:04 WARN Pod/dashboard-redirect-779874bd57-hpm6n: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:04 WARN Pod/maas-postgres-bb9d55498-k2hcr: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:04 WARN Pod/odh-model-controller-65fdb9c5c-4h5f5: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:04 WARN Pod/kserve-controller-manager-dd9f6cf9-zgfp2: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:04 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-qfcc6 2026/07/19 19:27:00 19:26:04 INFO ReplicaSet/dashboard-redirect-779874bd57: SuccessfulCreate - Created pod: dashboard-redirect-779874bd57-hpm6n 2026/07/19 19:27:00 19:26:04 INFO ReplicaSet/kuberay-operator-6586cdc875: SuccessfulCreate - Created pod: kuberay-operator-6586cdc875-gfgkd 2026/07/19 19:27:00 19:26:04 INFO ReplicaSet/odh-model-controller-65fdb9c5c: SuccessfulCreate - Created pod: odh-model-controller-65fdb9c5c-4h5f5 2026/07/19 19:27:00 19:26:04 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-zgfp2 2026/07/19 19:27:00 19:26:04 INFO ReplicaSet/maas-postgres-bb9d55498: SuccessfulCreate - Created pod: maas-postgres-bb9d55498-k2hcr 2026/07/19 19:27:00 19:26:03 WARN Pod/model-registry-operator-controller-manager-777cd8666c-7lv7j: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:03 WARN Pod/kubeflow-trainer-controller-manager-b5fd84b9c-qvkwb: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:03 WARN Pod/data-science-pipelines-operator-controller-manager-7bf7988pmrqn: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:03 WARN Pod/notebook-controller-deployment-647b7d4d45-5hpjg: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:26:03 INFO ReplicaSet/notebook-controller-deployment-647b7d4d45: SuccessfulCreate - Created pod: notebook-controller-deployment-647b7d4d45-5hpjg 2026/07/19 19:27:00 19:26:03 INFO ReplicaSet/model-registry-operator-controller-manager-777cd8666c: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-777cd8666c-7lv7j 2026/07/19 19:27:00 19:26:03 INFO ReplicaSet/kubeflow-trainer-controller-manager-b5fd84b9c: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-b5fd84b9c-qvkwb 2026/07/19 19:27:00 19:26:03 INFO ReplicaSet/data-science-pipelines-operator-controller-manager-7bf7988f59: SuccessfulCreate - Created pod: data-science-pipelines-operator-controller-manager-7bf7988pmrqn 2026/07/19 19:27:00 19:24:46 INFO Pod/odh-notebook-controller-manager-fd87fc88b-7qgxr: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/odh-notebook-controller-manager-fd87fc88b-7qgxr 2026/07/19 19:27:00 19:24:46 WARN Pod/kubeflow-trainer-controller-manager-b5fd84b9c-jxhsr: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 WARN Pod/model-serving-api-5679d6b4f-mf4bm: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 WARN Pod/odh-model-controller-65fdb9c5c-t9qxc: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 INFO Pod/model-registry-operator-controller-manager-777cd8666c-5wdlj: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/model-registry-operator-controller-manager-777cd8666c-5wdlj 2026/07/19 19:27:00 19:24:46 INFO Pod/dashboard-redirect-779874bd57-ggrcf: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/dashboard-redirect-779874bd57-ggrcf 2026/07/19 19:27:00 19:24:46 INFO Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-pi-d9ei7m74nvms7b57vg8g-driver 2026/07/19 19:27:00 19:24:46 WARN Pod/rhods-dashboard-6fbb47df4b-54kpk: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 WARN Pod/global-pull-secret-syncer-xh4wr: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 WARN Pod/konnectivity-agent-rhl8q: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 INFO Pod/rhods-dashboard-6fbb47df4b-6m7gx: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/rhods-dashboard-6fbb47df4b-6m7gx 2026/07/19 19:27:00 19:24:46 WARN Pod/maas-postgres-bb9d55498-gvlmh: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 WARN Pod/llmisvc-controller-manager-7b46b9fbbb-bjdll: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 WARN Pod/kube-apiserver-proxy-ip-10-0-133-216.ec2.internal: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 INFO Pod/spark-operator-controller-5dd7d48f87-7x4c9: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-7x4c9 2026/07/19 19:27:00 19:24:46 WARN Pod/spark-operator-controller-5dd7d48f87-q6lw8: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 INFO Pod/kuberay-operator-6586cdc875-pcq4t: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/kuberay-operator-6586cdc875-pcq4t 2026/07/19 19:27:00 19:24:46 WARN Pod/notebook-controller-deployment-647b7d4d45-btjxp: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 WARN Pod/kubeflow-training-operator-54786944d6-fj2r8: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 INFO Pod/dashboard-redirect-779874bd57-knjcn: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/dashboard-redirect-779874bd57-knjcn 2026/07/19 19:27:00 19:24:46 INFO Pod/spark-operator-webhook-5f8768dbfd-4w5s8: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-4w5s8 2026/07/19 19:27:00 19:24:46 WARN Pod/trustyai-service-operator-controller-manager-5dccbd95db-qvdqb: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 WARN Pod/kserve-controller-manager-dd9f6cf9-rh9qs: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 WARN Pod/spark-operator-webhook-5f8768dbfd-p9xm2: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 WARN Pod/feast-operator-controller-manager-7b7cd769f7-8lxtw: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:46 WARN Pod/data-science-pipelines-operator-controller-manager-7bf7988pzj4b: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:44 WARN Pod/spark-pi-d7dcb59f7bd45831-exec-1: FailedScheduling - 0/3 nodes are available: 1 Insufficient cpu, 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling. 2026/07/19 19:27:00 19:24:40 WARN Pod/rhods-dashboard-6fbb47df4b-6m7gx: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:40 WARN Pod/global-pull-secret-syncer-nc76j: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:40 WARN Pod/kube-apiserver-proxy-ip-10-0-141-80.ec2.internal: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:40 WARN Pod/konnectivity-agent-xvtnl: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:40 WARN Pod/odh-notebook-controller-manager-fd87fc88b-7qgxr: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:40 WARN Pod/dashboard-redirect-779874bd57-ggrcf: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:34 WARN Pod/spark-pi-d7dcb59f7bd45831-exec-1: FailedScheduling - 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod. 2026/07/19 19:27:00 19:24:34 WARN Pod/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:34 WARN Pod/konnectivity-agent-dqrmr: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:34 WARN Pod/spark-operator-webhook-5f8768dbfd-4w5s8: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:34 WARN Pod/model-registry-operator-controller-manager-777cd8666c-5wdlj: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:34 WARN Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:34 WARN Pod/kuberay-operator-6586cdc875-pcq4t: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:34 WARN Pod/global-pull-secret-syncer-jvwzt: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:34 WARN Pod/spark-operator-controller-5dd7d48f87-7x4c9: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:24:34 WARN Pod/dashboard-redirect-779874bd57-knjcn: NodeNotReady - Node is not ready 2026/07/19 19:27:00 19:23:51 WARN Pod/spark-pi-d7dcb59f7bd45831-exec-1: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/07/19 19:27:00 19:22:24 WARN Pod/spark-pi-d7dcb59f7bd45831-exec-1: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/07/19 19:27:00 19:22:24 INFO SparkApplication/spark-pi-d9ei7m74nvms7b57vg8g: SparkExecutorPending - Executor spark-pi-d7dcb59f7bd45831-exec-1 is pending 2026/07/19 19:27:00 19:22:16 INFO SparkApplication/spark-pi-d9ei7m74nvms7b57vg8g: SparkDriverRunning - Driver spark-pi-d9ei7m74nvms7b57vg8g-driver is running 2026/07/19 19:27:00 19:22:15 INFO Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: Started - Started container spark-kubernetes-driver 2026/07/19 19:27:00 19:22:15 INFO Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: Created - Created container: spark-kubernetes-driver 2026/07/19 19:27:00 19:22:15 INFO Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/19 19:27:00 19:22:15 INFO Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: AddedInterface - Add eth0 [10.132.0.150/23] from ovn-kubernetes 2026/07/19 19:27:00 19:22:14 INFO Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-d9ei7m74nvms7b57vg8g-driver to ip-10-0-135-209.ec2.internal 2026/07/19 19:27:00 19:22:14 INFO SparkApplication/spark-pi-d9ei7m74nvms7b57vg8g: SparkApplicationSubmitted - SparkApplication spark-pi-d9ei7m74nvms7b57vg8g was submitted successfully 2026/07/19 19:27:00 19:22:14 WARN Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d9ei7m74nvms7b57vg8g-d5ae6e9f7bd42361-driver-podspec-conf-map" not found 2026/07/19 19:27:00 19:22:14 WARN Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-d7dcd89f7bd434dc-conf-map" not found 2026/07/19 19:27:00 === RESOURCE QUOTAS === 2026/07/19 19:27:00 No resource quotas found in monitored namespaces 2026/07/19 19:27:00 Classification for group 1: infrastructure/storage (code=1006, confidence=medium) 2026/07/19 19:27:00 Evidence: event Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d9ei7m74nvms7b57vg8g-d5ae6e9f7bd42361-driver-podspec-conf-map" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: FailedMount - MountVolume.SetUp failed for volume \"pod-template-volume\" : configmap \"spark-pi-d9ei7m74nvms7b57vg8g-d5ae6e9f7bd42361-driver-podspec-conf-map\" not found"],"confidence":"medium"} 2026/07/19 19:27:00 Diagnostics complete for failed test: group 1 controller_test.go:753: Stopping: group 1 test failed. 2026/07/19 19:27:00 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/19 19:27:00 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/19 19:27:00 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (370.56s) === RUN TestOdhOperator 2026/07/19 19:20:40 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/19 19:27:00 TEST FAILURE DETECTED: components 2026/07/19 19:27:00 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/19 19:27:00 clusterhealth: node metrics unavailable (Metrics Server may not be installed): the server is currently unable to handle the request (get nodes.metrics.k8s.io) 2026/07/19 19:27:01 === CLUSTER STATE === 2026/07/19 19:27:01 Failed to collect node data: unhealthy nodes: ip-10-0-141-80.ec2.internal (Ready=Unknown) 2026/07/19 19:27:01 === DEPLOYMENTS === 2026/07/19 19:27:01 Failed to collect deployment data: deployments not ready: redhat-ods-applications/dashboard-redirect (0/2), redhat-ods-applications/data-science-pipelines-operator-controller-manager (0/1), redhat-ods-applications/feast-operator-controller-manager (0/1), redhat-ods-applications/kserve-controller-manager (0/1), redhat-ods-applications/kubeflow-trainer-controller-manager (0/1), redhat-ods-applications/kubeflow-training-operator (0/1), redhat-ods-applications/kuberay-operator (0/1), redhat-ods-applications/llmisvc-controller-manager (0/1), redhat-ods-applications/maas-postgres (0/1), redhat-ods-applications/model-registry-operator-controller-manager (0/1), redhat-ods-applications/model-serving-api (0/1), redhat-ods-applications/notebook-controller-deployment (0/1), redhat-ods-applications/odh-model-controller (0/1), redhat-ods-applications/odh-notebook-controller-manager (0/1), redhat-ods-applications/rhods-dashboard (0/2), redhat-ods-applications/spark-operator-controller (0/2), redhat-ods-applications/spark-operator-webhook (0/2), redhat-ods-applications/trustyai-service-operator-controller-manager (0/1) 2026/07/19 19:27:01 === PODS === 2026/07/19 19:27:01 Failed to collect pod data: redhat-ods-applications/dashboard-redirect-779874bd57-hpm6n: phase=Pending; redhat-ods-applications/data-science-pipelines-operator-controller-manager-7bf7988pmrqn: phase=Pending; redhat-ods-applications/feast-operator-controller-manager-7b7cd769f7-5t994: phase=Pending; redhat-ods-applications/kserve-controller-manager-dd9f6cf9-zgfp2: phase=Pending; redhat-ods-applications/kubeflow-trainer-controller-manager-b5fd84b9c-qvkwb: phase=Pending; redhat-ods-applications/kubeflow-training-operator-54786944d6-8cz6d: phase=Pending; redhat-ods-applications/kuberay-operator-6586cdc875-gfgkd: phase=Pending; redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-sn7lz: phase=Pending; redhat-ods-applications/maas-postgres-bb9d55498-k2hcr: phase=Pending; redhat-ods-applications/model-registry-operator-controller-manager-777cd8666c-7lv7j: phase=Pending; redhat-ods-applications/model-serving-api-5679d6b4f-n77rx: phase=Pending; redhat-ods-applications/notebook-controller-deployment-647b7d4d45-5hpjg: phase=Pending; redhat-ods-applications/odh-model-controller-65fdb9c5c-4h5f5: phase=Pending; redhat-ods-applications/rhods-dashboard-6fbb47df4b-56llp: phase=Pending; redhat-ods-applications/spark-operator-controller-5dd7d48f87-2kdj4: phase=Pending; redhat-ods-applications/spark-operator-controller-5dd7d48f87-qfcc6: phase=Pending; redhat-ods-applications/spark-operator-webhook-5f8768dbfd-4x44h: phase=Pending; redhat-ods-applications/spark-operator-webhook-5f8768dbfd-cdn48: phase=Pending; redhat-ods-applications/trustyai-service-operator-controller-manager-5dccbd95db-s5jlt: phase=Pending 2026/07/19 19:27:01 === OPERATOR STATUS === 2026/07/19 19:27:01 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-5d8f6b8fb-dhdgv: Pending; operator pod rhods-operator-5d8f6b8fb-m4d8c: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-p6djc: Pending; dependent jobset: deployment 0/1 ready; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; pod cert-manager-operator-controller-manager-55c444bf86-dkb8m: Pending; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-z2vcf: Pending 2026/07/19 19:27:01 === DSCI STATUS === 2026/07/19 19:27:01 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/19 19:27:01 === DSC STATUS === 2026/07/19 19:27:01 DSC default-dsc: 2026/07/19 19:27:01 Ready: True - 2026/07/19 19:27:01 ProvisioningSucceeded: True - 2026/07/19 19:27:01 ComponentsReady: True - 2026/07/19 19:27:01 AIPipelinesReady: True - 2026/07/19 19:27:01 DashboardReady: True - 2026/07/19 19:27:01 FeastOperatorReady: True - 2026/07/19 19:27:01 KserveLLMInferenceServiceDependencies: True - 2026/07/19 19:27:01 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/19 19:27:01 KserveReady: True - 2026/07/19 19:27:01 KueueReady: False - Component ManagementState is set to Removed 2026/07/19 19:27:01 LLM-D-WVADependencies: True - 2026/07/19 19:27:01 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/19 19:27:01 ModelControllerReady: True - 2026/07/19 19:27:01 ModelRegistryReady: True - 2026/07/19 19:27:01 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/19 19:27:01 OGXReady: False - Component ManagementState is set to Removed 2026/07/19 19:27:01 RayReady: True - 2026/07/19 19:27:01 SparkOperatorReady: True - 2026/07/19 19:27:01 TrainerReady: True - 2026/07/19 19:27:01 TrainingOperatorReady: True - 2026/07/19 19:27:01 TrustyAIReady: True - 2026/07/19 19:27:01 WorkbenchesReady: True - 2026/07/19 19:27:01 === RECENT EVENTS === 2026/07/19 19:27:01 19:26:06 WARN Pod/rhods-dashboard-6fbb47df4b-56llp: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:06 WARN Pod/feast-operator-controller-manager-7b7cd769f7-5t994: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:06 WARN Pod/kubeflow-training-operator-54786944d6-8cz6d: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:06 WARN Pod/spark-operator-webhook-5f8768dbfd-4x44h: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:06 INFO ReplicaSet/feast-operator-controller-manager-7b7cd769f7: SuccessfulCreate - Created pod: feast-operator-controller-manager-7b7cd769f7-5t994 2026/07/19 19:27:01 19:26:06 INFO ReplicaSet/kubeflow-training-operator-54786944d6: SuccessfulCreate - Created pod: kubeflow-training-operator-54786944d6-8cz6d 2026/07/19 19:27:01 19:26:06 INFO ReplicaSet/rhods-dashboard-6fbb47df4b: SuccessfulCreate - Created pod: rhods-dashboard-6fbb47df4b-56llp 2026/07/19 19:27:01 19:26:05 WARN Pod/trustyai-service-operator-controller-manager-5dccbd95db-s5jlt: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:05 WARN Pod/model-serving-api-5679d6b4f-n77rx: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:05 WARN Pod/llmisvc-controller-manager-7b46b9fbbb-sn7lz: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:05 WARN Pod/spark-operator-webhook-5f8768dbfd-cdn48: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:05 WARN Pod/spark-operator-controller-5dd7d48f87-2kdj4: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:05 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-4x44h 2026/07/19 19:27:01 19:26:05 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-sn7lz 2026/07/19 19:27:01 19:26:05 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-n77rx 2026/07/19 19:27:01 19:26:05 WARN Service/spark-operator-webhook-svc: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/spark-operator-webhook-svc: skipping Pod spark-operator-webhook-5f8768dbfd-4w5s8 for Service redhat-ods-applications/spark-operator-webhook-svc: Node ip-10-0-135-209.ec2.internal Not Found 2026/07/19 19:27:01 19:26:05 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-cdn48 2026/07/19 19:27:01 19:26:05 INFO ReplicaSet/trustyai-service-operator-controller-manager-5dccbd95db: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-5dccbd95db-s5jlt 2026/07/19 19:27:01 19:26:05 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-2kdj4 2026/07/19 19:27:01 19:26:04 WARN Pod/spark-operator-controller-5dd7d48f87-qfcc6: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:04 WARN Pod/kuberay-operator-6586cdc875-gfgkd: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:04 WARN Pod/dashboard-redirect-779874bd57-hpm6n: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:04 WARN Pod/maas-postgres-bb9d55498-k2hcr: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:04 WARN Pod/odh-model-controller-65fdb9c5c-4h5f5: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:04 WARN Pod/kserve-controller-manager-dd9f6cf9-zgfp2: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:04 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-qfcc6 2026/07/19 19:27:01 19:26:04 INFO ReplicaSet/dashboard-redirect-779874bd57: SuccessfulCreate - Created pod: dashboard-redirect-779874bd57-hpm6n 2026/07/19 19:27:01 19:26:04 INFO ReplicaSet/kuberay-operator-6586cdc875: SuccessfulCreate - Created pod: kuberay-operator-6586cdc875-gfgkd 2026/07/19 19:27:01 19:26:04 INFO ReplicaSet/odh-model-controller-65fdb9c5c: SuccessfulCreate - Created pod: odh-model-controller-65fdb9c5c-4h5f5 2026/07/19 19:27:01 19:26:04 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-zgfp2 2026/07/19 19:27:01 19:26:04 INFO ReplicaSet/maas-postgres-bb9d55498: SuccessfulCreate - Created pod: maas-postgres-bb9d55498-k2hcr 2026/07/19 19:27:01 19:26:03 WARN Pod/model-registry-operator-controller-manager-777cd8666c-7lv7j: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:03 WARN Pod/kubeflow-trainer-controller-manager-b5fd84b9c-qvkwb: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:03 WARN Pod/data-science-pipelines-operator-controller-manager-7bf7988pmrqn: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:03 WARN Pod/notebook-controller-deployment-647b7d4d45-5hpjg: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:26:03 INFO ReplicaSet/notebook-controller-deployment-647b7d4d45: SuccessfulCreate - Created pod: notebook-controller-deployment-647b7d4d45-5hpjg 2026/07/19 19:27:01 19:26:03 INFO ReplicaSet/model-registry-operator-controller-manager-777cd8666c: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-777cd8666c-7lv7j 2026/07/19 19:27:01 19:26:03 INFO ReplicaSet/kubeflow-trainer-controller-manager-b5fd84b9c: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-b5fd84b9c-qvkwb 2026/07/19 19:27:01 19:26:03 INFO ReplicaSet/data-science-pipelines-operator-controller-manager-7bf7988f59: SuccessfulCreate - Created pod: data-science-pipelines-operator-controller-manager-7bf7988pmrqn 2026/07/19 19:27:01 19:24:46 INFO Pod/odh-notebook-controller-manager-fd87fc88b-7qgxr: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/odh-notebook-controller-manager-fd87fc88b-7qgxr 2026/07/19 19:27:01 19:24:46 WARN Pod/kubeflow-trainer-controller-manager-b5fd84b9c-jxhsr: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 WARN Pod/model-serving-api-5679d6b4f-mf4bm: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 WARN Pod/odh-model-controller-65fdb9c5c-t9qxc: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 INFO Pod/model-registry-operator-controller-manager-777cd8666c-5wdlj: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/model-registry-operator-controller-manager-777cd8666c-5wdlj 2026/07/19 19:27:01 19:24:46 INFO Pod/dashboard-redirect-779874bd57-ggrcf: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/dashboard-redirect-779874bd57-ggrcf 2026/07/19 19:27:01 19:24:46 INFO Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-pi-d9ei7m74nvms7b57vg8g-driver 2026/07/19 19:27:01 19:24:46 WARN Pod/rhods-dashboard-6fbb47df4b-54kpk: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 WARN Pod/global-pull-secret-syncer-xh4wr: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 WARN Pod/konnectivity-agent-rhl8q: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 INFO Pod/rhods-dashboard-6fbb47df4b-6m7gx: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/rhods-dashboard-6fbb47df4b-6m7gx 2026/07/19 19:27:01 19:24:46 WARN Pod/maas-postgres-bb9d55498-gvlmh: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 WARN Pod/llmisvc-controller-manager-7b46b9fbbb-bjdll: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 WARN Pod/kube-apiserver-proxy-ip-10-0-133-216.ec2.internal: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 INFO Pod/spark-operator-controller-5dd7d48f87-7x4c9: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-7x4c9 2026/07/19 19:27:01 19:24:46 WARN Pod/spark-operator-controller-5dd7d48f87-q6lw8: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 INFO Pod/kuberay-operator-6586cdc875-pcq4t: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/kuberay-operator-6586cdc875-pcq4t 2026/07/19 19:27:01 19:24:46 WARN Pod/notebook-controller-deployment-647b7d4d45-btjxp: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 WARN Pod/kubeflow-training-operator-54786944d6-fj2r8: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 INFO Pod/dashboard-redirect-779874bd57-knjcn: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/dashboard-redirect-779874bd57-knjcn 2026/07/19 19:27:01 19:24:46 INFO Pod/spark-operator-webhook-5f8768dbfd-4w5s8: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-4w5s8 2026/07/19 19:27:01 19:24:46 WARN Pod/trustyai-service-operator-controller-manager-5dccbd95db-qvdqb: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 WARN Pod/kserve-controller-manager-dd9f6cf9-rh9qs: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 WARN Pod/spark-operator-webhook-5f8768dbfd-p9xm2: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 WARN Pod/feast-operator-controller-manager-7b7cd769f7-8lxtw: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:46 WARN Pod/data-science-pipelines-operator-controller-manager-7bf7988pzj4b: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:44 WARN Pod/spark-pi-d7dcb59f7bd45831-exec-1: FailedScheduling - 0/3 nodes are available: 1 Insufficient cpu, 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling. 2026/07/19 19:27:01 19:24:40 WARN Pod/rhods-dashboard-6fbb47df4b-6m7gx: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:40 WARN Pod/global-pull-secret-syncer-nc76j: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:40 WARN Pod/kube-apiserver-proxy-ip-10-0-141-80.ec2.internal: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:40 WARN Pod/konnectivity-agent-xvtnl: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:40 WARN Pod/odh-notebook-controller-manager-fd87fc88b-7qgxr: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:40 WARN Pod/dashboard-redirect-779874bd57-ggrcf: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:34 WARN Pod/spark-pi-d7dcb59f7bd45831-exec-1: FailedScheduling - 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod. 2026/07/19 19:27:01 19:24:34 WARN Pod/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:34 WARN Pod/konnectivity-agent-dqrmr: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:34 WARN Pod/spark-operator-webhook-5f8768dbfd-4w5s8: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:34 WARN Pod/model-registry-operator-controller-manager-777cd8666c-5wdlj: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:34 WARN Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:34 WARN Pod/kuberay-operator-6586cdc875-pcq4t: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:34 WARN Pod/global-pull-secret-syncer-jvwzt: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:34 WARN Pod/spark-operator-controller-5dd7d48f87-7x4c9: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:24:34 WARN Pod/dashboard-redirect-779874bd57-knjcn: NodeNotReady - Node is not ready 2026/07/19 19:27:01 19:23:51 WARN Pod/spark-pi-d7dcb59f7bd45831-exec-1: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/07/19 19:27:01 19:22:24 WARN Pod/spark-pi-d7dcb59f7bd45831-exec-1: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/07/19 19:27:01 19:22:24 INFO SparkApplication/spark-pi-d9ei7m74nvms7b57vg8g: SparkExecutorPending - Executor spark-pi-d7dcb59f7bd45831-exec-1 is pending 2026/07/19 19:27:01 19:22:16 INFO SparkApplication/spark-pi-d9ei7m74nvms7b57vg8g: SparkDriverRunning - Driver spark-pi-d9ei7m74nvms7b57vg8g-driver is running 2026/07/19 19:27:01 19:22:15 INFO Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: Started - Started container spark-kubernetes-driver 2026/07/19 19:27:01 19:22:15 INFO Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: Created - Created container: spark-kubernetes-driver 2026/07/19 19:27:01 19:22:15 INFO Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/19 19:27:01 19:22:15 INFO Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: AddedInterface - Add eth0 [10.132.0.150/23] from ovn-kubernetes 2026/07/19 19:27:01 19:22:14 INFO Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-d9ei7m74nvms7b57vg8g-driver to ip-10-0-135-209.ec2.internal 2026/07/19 19:27:01 19:22:14 INFO SparkApplication/spark-pi-d9ei7m74nvms7b57vg8g: SparkApplicationSubmitted - SparkApplication spark-pi-d9ei7m74nvms7b57vg8g was submitted successfully 2026/07/19 19:27:01 19:22:14 WARN Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d9ei7m74nvms7b57vg8g-d5ae6e9f7bd42361-driver-podspec-conf-map" not found 2026/07/19 19:27:01 19:22:14 WARN Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-d7dcd89f7bd434dc-conf-map" not found 2026/07/19 19:27:01 === RESOURCE QUOTAS === 2026/07/19 19:27:01 No resource quotas found in monitored namespaces 2026/07/19 19:27:01 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/07/19 19:27:01 Evidence: event Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d9ei7m74nvms7b57vg8g-d5ae6e9f7bd42361-driver-podspec-conf-map" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/spark-pi-d9ei7m74nvms7b57vg8g-driver: FailedMount - MountVolume.SetUp failed for volume \"pod-template-volume\" : configmap \"spark-pi-d9ei7m74nvms7b57vg8g-d5ae6e9f7bd42361-driver-podspec-conf-map\" not found"],"confidence":"medium"} 2026/07/19 19:27:01 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/19 19:27:01 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/19 19:27:01 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/19 19:27:01 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (381.33s)