=== RUN TestOdhOperator/components/group_1/sparkoperator/Validate_SparkPi_workload_execution helper_test.go:150: Creating SparkApplication spark-pi-d9eskj6429es7b645uvg in namespace redhat-ods-applications helper_test.go:150: Waiting for SparkApplication spark-pi-d9eskj6429es7b645uvg to complete sparkoperator_test.go:72: Cleaning up SparkApplication spark-pi-d9eskj6429es7b645uvg testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/sparkoperator/Validate_SparkPi_workload_execution (300.06s) === RUN TestOdhOperator/components/group_1/sparkoperator === PAUSE TestOdhOperator/components/group_1/sparkoperator === CONT TestOdhOperator/components/group_1/sparkoperator 2026-07-20T07:11:01Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.25"}} test_context_test.go:223: Timed out after 300.001s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: SparkApplication spark-pi-d9eskj6429es7b645uvg did not complete successfully Expected <string>: map[apiVersion:sparkoperator.k8s.io/v1beta2 kind:SparkApplication metadata:map[creationTimestamp:2026-07-20T07:12:12Z generation:1 managedFields:[map[apiVersion:sparkoperator.k8s.io/v1beta2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[.:map[] f:arguments:map[] f:driver:map[.:map[] f:coreLimit:map[] f:cores:map[] f:labels:map[.:map[] f:version:map[]] f:memory:map[] f:securityContext:map[] f:serviceAccount:map[] f:volumeMounts:map[]] f:executor:map[.:map[] f:cores:map[] f:instances:map[] f:labels:map[.:map[] f:version:map[]] f:memory:map[] f:securityContext:map[] f:volumeMounts:map[]] f:image:map[] f:imagePullPolicy:map[] f:mainApplicationFile:map[] f:mainClass:map[] f:mode:map[] f:restartPolicy:map[.:map[] f:type:map[]] f:sparkConf:map[.:map[] f:spark.blockManager.port:map[] f:spark.driver.blockManager.port:map[] f:spark.driver.port:map[] f:spark.port.maxRetries:map[]] f:sparkVersion:map[] f:type:map[] f:volumes:map[]]] manager:e2e-tests operation:Update time:2026-07-20T07:12:12Z] map[apiVersion:sparkoperator.k8s.io/v1beta2 fieldsType:FieldsV1 fieldsV1:map[f:status:map[.:map[] f:applicationState:map[.:map[] f:state:map[]] f:driverInfo:map[.:map[] f:podName:map[] f:webUIAddress:map[] f:webUIPort:map[] f:webUIServiceName:map[]] f:executionAttempts:map[] f:lastSubmissionAttemptTime:map[] f:sparkApplicationId:map[] f:submissionAttempts:map[] f:submissionID:map[] f:terminationTime:map[]]] manager:spark-operator operation:Update subresource:status time:2026-07-20T07:12:26Z]] name:spark-pi-d9eskj6429es7b645uvg namespace:redhat-ods-applications resourceVersion:144168 uid:a160e828-778f-487b-a791-32255b4c558d] spec:map[arguments:[1000] deps:map[] driver:map[coreLimit:1200m cores:1 labels:map[version:4.0.1] memory:512m securityContext:map[] serviceAccount:spark-operator-spark volumeMounts:[map[mountPath:/opt/spark/work-dir name:spark-work-dir]]] executor:map[cores:1 instances:1 labels:map[version:4.0.1] memory:512m securityContext:map[] volumeMounts:[map[mountPath:/opt/spark/work-dir name:spark-work-dir]]] image:quay.io/opendatahub/data-processing:Spark-v4.0.1 imagePullPolicy:IfNotPresent mainApplicationFile:local:///opt/spark/examples/jars/spark-examples_2.13-4.0.1.jar mainClass:org.apache.spark.examples.SparkPi mode:cluster restartPolicy:map[type:Never] sparkConf:map[spark.blockManager.port:8081 spark.driver.blockManager.port:8082 spark.driver.port:8080 spark.port.maxRetries:0] sparkVersion:4.0.1 type:Scala volumes:[map[emptyDir:map[] name:spark-work-dir]]] status:map[applicationState:map[state:SUBMITTED] driverInfo:map[podName:spark-pi-d9eskj6429es7b645uvg-driver webUIAddress:172.31.69.88:4040 webUIPort:4040 webUIServiceName:spark-pi-d9eskj6429es7b645uvg-ui-svc] executionAttempts:1 lastSubmissionAttemptTime:2026-07-20T07:12:12Z sparkApplicationId:spark-00fd8a7cf88b46f2a844bba0d217707d submissionAttempts:1 submissionID:5f3a2929-54e0-46da-af87-3c937cf201aa terminationTime:<nil>]] to match expression <string>: .status.applicationState.state == "COMPLETED" --- FAIL: TestOdhOperator/components/group_1/sparkoperator (370.46s) === RUN TestOdhOperator/components/group_1 controller_test.go:753: Skipping tests for components/aigateway controller_test.go:753: Skipping tests for components/dashboard controller_test.go:753: Skipping tests for components/datasciencepipelines controller_test.go:753: Skipping tests for components/feastoperator controller_test.go:753: Skipping tests for components/kserve controller_test.go:753: Skipping tests for components/modelregistry controller_test.go:753: Skipping tests for components/ogx controller_test.go:753: Skipping tests for components/ray controller_test.go:753: Skipping tests for components/trainer controller_test.go:753: Skipping tests for components/trainingoperator controller_test.go:753: Skipping tests for components/workbenches --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components 2026/07/20 07:17:12 TEST FAILURE DETECTED: group 1 2026/07/20 07:17:12 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/20 07:17:12 === CLUSTER STATE === 2026/07/20 07:17:12 Node: ip-10-0-134-85.ec2.internal 2026/07/20 07:17:12 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/20 07:17:12 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/20 07:17:12 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/20 07:17:12 Condition Ready: True - kubelet is posting ready status 2026/07/20 07:17:12 Resources: 7500m CPU, 31236224Ki memory allocatable (of 8000m CPU, 32387200Ki memory capacity) 2026/07/20 07:17:12 Node: ip-10-0-142-103.ec2.internal 2026/07/20 07:17:12 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/20 07:17:12 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/20 07:17:12 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/20 07:17:12 Condition Ready: True - kubelet is posting ready status 2026/07/20 07:17:12 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/20 07:17:12 Node: ip-10-0-143-14.ec2.internal 2026/07/20 07:17:12 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/20 07:17:12 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/20 07:17:12 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/20 07:17:12 Condition Ready: True - kubelet is posting ready status 2026/07/20 07:17:12 Resources: 7500m CPU, 31236224Ki memory allocatable (of 8000m CPU, 32387200Ki memory capacity) 2026/07/20 07:17:12 === DEPLOYMENTS === 2026/07/20 07:17:12 Namespace: redhat-ods-applications 2026/07/20 07:17:12 Deployment dashboard-redirect: 2/2 (READY) 2026/07/20 07:17:12 Deployment data-science-pipelines-operator-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Deployment feast-operator-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Deployment kuberay-operator: 1/1 (READY) 2026/07/20 07:17:12 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Deployment maas-postgres: 1/1 (READY) 2026/07/20 07:17:12 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Deployment model-serving-api: 1/1 (READY) 2026/07/20 07:17:12 Deployment notebook-controller-deployment: 1/1 (READY) 2026/07/20 07:17:12 Deployment odh-model-controller: 1/1 (READY) 2026/07/20 07:17:12 Deployment odh-notebook-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Deployment rhods-dashboard: 2/2 (READY) 2026/07/20 07:17:12 Deployment spark-operator-controller: 2/2 (READY) 2026/07/20 07:17:12 Deployment spark-operator-webhook: 2/2 (READY) 2026/07/20 07:17:12 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Namespace: kube-system 2026/07/20 07:17:12 === PODS === 2026/07/20 07:17:12 Namespace redhat-ods-applications: no problematic pods found 2026/07/20 07:17:12 Namespace kube-system: no problematic pods found 2026/07/20 07:17:12 === OPERATOR STATUS === 2026/07/20 07:17:12 Operator deployment: 3/3 ready 2026/07/20 07:17:12 === DSCI STATUS === 2026/07/20 07:17:12 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/20 07:17:12 === DSC STATUS === 2026/07/20 07:17:12 DSC default-dsc: 2026/07/20 07:17:12 Ready: True - 2026/07/20 07:17:12 ProvisioningSucceeded: True - 2026/07/20 07:17:12 ComponentsReady: True - 2026/07/20 07:17:12 AIPipelinesReady: True - 2026/07/20 07:17:12 DashboardReady: True - 2026/07/20 07:17:12 FeastOperatorReady: True - 2026/07/20 07:17:12 KserveLLMInferenceServiceDependencies: True - 2026/07/20 07:17:12 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/20 07:17:12 KserveReady: True - 2026/07/20 07:17:12 KueueReady: False - Component ManagementState is set to Removed 2026/07/20 07:17:12 LLM-D-WVADependencies: True - 2026/07/20 07:17:12 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/20 07:17:12 ModelControllerReady: True - 2026/07/20 07:17:12 ModelRegistryReady: True - 2026/07/20 07:17:12 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/20 07:17:12 OGXReady: False - Component ManagementState is set to Removed 2026/07/20 07:17:12 RayReady: True - 2026/07/20 07:17:12 SparkOperatorReady: True - 2026/07/20 07:17:12 TrainerReady: False - Component ManagementState is set to Removed 2026/07/20 07:17:12 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/20 07:17:12 TrustyAIReady: True - 2026/07/20 07:17:12 WorkbenchesReady: True - 2026/07/20 07:17:12 === RECENT EVENTS === 2026/07/20 07:17:12 07:12:26 WARN Pod/spark-pi-d9eskj6429es7b645uvg-driver: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/07/20 07:17:12 07:12:26 INFO SparkApplication/spark-pi-d9eskj6429es7b645uvg: SparkApplicationSubmitted - SparkApplication spark-pi-d9eskj6429es7b645uvg was submitted successfully 2026/07/20 07:17:12 === RESOURCE QUOTAS === 2026/07/20 07:17:12 No resource quotas found in monitored namespaces 2026/07/20 07:17:12 Classification for group 1: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/20 07:17:12 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/20 07:17:12 Diagnostics complete for failed test: group 1 controller_test.go:753: Stopping: group 1 test failed. 2026/07/20 07:17:12 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/20 07:17:12 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/20 07:17:12 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (370.62s) === RUN TestOdhOperator 2026/07/20 07:10:51 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/20 07:17:12 TEST FAILURE DETECTED: components 2026/07/20 07:17:12 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/20 07:17:12 === CLUSTER STATE === 2026/07/20 07:17:12 Node: ip-10-0-134-85.ec2.internal 2026/07/20 07:17:12 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/20 07:17:12 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/20 07:17:12 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/20 07:17:12 Condition Ready: True - kubelet is posting ready status 2026/07/20 07:17:12 Resources: 7500m CPU, 31236224Ki memory allocatable (of 8000m CPU, 32387200Ki memory capacity) 2026/07/20 07:17:12 Node: ip-10-0-142-103.ec2.internal 2026/07/20 07:17:12 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/20 07:17:12 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/20 07:17:12 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/20 07:17:12 Condition Ready: True - kubelet is posting ready status 2026/07/20 07:17:12 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/20 07:17:12 Node: ip-10-0-143-14.ec2.internal 2026/07/20 07:17:12 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/20 07:17:12 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/20 07:17:12 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/20 07:17:12 Condition Ready: True - kubelet is posting ready status 2026/07/20 07:17:12 Resources: 7500m CPU, 31236224Ki memory allocatable (of 8000m CPU, 32387200Ki memory capacity) 2026/07/20 07:17:12 === DEPLOYMENTS === 2026/07/20 07:17:12 Namespace: redhat-ods-applications 2026/07/20 07:17:12 Deployment dashboard-redirect: 2/2 (READY) 2026/07/20 07:17:12 Deployment data-science-pipelines-operator-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Deployment feast-operator-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Deployment kuberay-operator: 1/1 (READY) 2026/07/20 07:17:12 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Deployment maas-postgres: 1/1 (READY) 2026/07/20 07:17:12 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Deployment model-serving-api: 1/1 (READY) 2026/07/20 07:17:12 Deployment notebook-controller-deployment: 1/1 (READY) 2026/07/20 07:17:12 Deployment odh-model-controller: 1/1 (READY) 2026/07/20 07:17:12 Deployment odh-notebook-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Deployment rhods-dashboard: 2/2 (READY) 2026/07/20 07:17:12 Deployment spark-operator-controller: 2/2 (READY) 2026/07/20 07:17:12 Deployment spark-operator-webhook: 2/2 (READY) 2026/07/20 07:17:12 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/20 07:17:12 Namespace: kube-system 2026/07/20 07:17:12 === PODS === 2026/07/20 07:17:12 Namespace redhat-ods-applications: no problematic pods found 2026/07/20 07:17:12 Namespace kube-system: no problematic pods found 2026/07/20 07:17:12 === OPERATOR STATUS === 2026/07/20 07:17:12 Operator deployment: 3/3 ready 2026/07/20 07:17:12 === DSCI STATUS === 2026/07/20 07:17:12 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/20 07:17:12 === DSC STATUS === 2026/07/20 07:17:12 DSC default-dsc: 2026/07/20 07:17:12 Ready: True - 2026/07/20 07:17:12 ProvisioningSucceeded: True - 2026/07/20 07:17:12 ComponentsReady: True - 2026/07/20 07:17:12 AIPipelinesReady: True - 2026/07/20 07:17:12 DashboardReady: True - 2026/07/20 07:17:12 FeastOperatorReady: True - 2026/07/20 07:17:12 KserveLLMInferenceServiceDependencies: True - 2026/07/20 07:17:12 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/20 07:17:12 KserveReady: True - 2026/07/20 07:17:12 KueueReady: False - Component ManagementState is set to Removed 2026/07/20 07:17:12 LLM-D-WVADependencies: True - 2026/07/20 07:17:12 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/20 07:17:12 ModelControllerReady: True - 2026/07/20 07:17:12 ModelRegistryReady: True - 2026/07/20 07:17:12 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/20 07:17:12 OGXReady: False - Component ManagementState is set to Removed 2026/07/20 07:17:12 RayReady: True - 2026/07/20 07:17:12 SparkOperatorReady: True - 2026/07/20 07:17:12 TrainerReady: False - Component ManagementState is set to Removed 2026/07/20 07:17:12 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/20 07:17:12 TrustyAIReady: True - 2026/07/20 07:17:12 WorkbenchesReady: True - 2026/07/20 07:17:12 === RECENT EVENTS === 2026/07/20 07:17:12 07:12:26 WARN Pod/spark-pi-d9eskj6429es7b645uvg-driver: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/07/20 07:17:12 07:12:26 INFO SparkApplication/spark-pi-d9eskj6429es7b645uvg: SparkApplicationSubmitted - SparkApplication spark-pi-d9eskj6429es7b645uvg was submitted successfully 2026/07/20 07:17:12 === RESOURCE QUOTAS === 2026/07/20 07:17:12 No resource quotas found in monitored namespaces 2026/07/20 07:17:12 Classification for components: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/20 07:17:12 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/20 07:17:12 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/20 07:17:12 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/20 07:17:12 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/20 07:17:12 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (381.51s)