=== RUN TestOdhOperator/components/group_1/sparkoperator/Validate_SparkPi_workload_execution helper_test.go:150: Creating SparkApplication spark-pi-dakm61v7fmuc7nkvk3d0 in namespace redhat-ods-applications helper_test.go:150: Waiting for SparkApplication spark-pi-dakm61v7fmuc7nkvk3d0 to complete sparkoperator_test.go:72: Cleaning up SparkApplication spark-pi-dakm61v7fmuc7nkvk3d0 testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/sparkoperator/Validate_SparkPi_workload_execution (300.26s) === RUN TestOdhOperator/components/group_1/sparkoperator === PAUSE TestOdhOperator/components/group_1/sparkoperator === CONT TestOdhOperator/components/group_1/sparkoperator 2026-09-15T15:22:40Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.31"}} test_context_test.go:223: Timed out after 300.009s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: SparkApplication spark-pi-dakm61v7fmuc7nkvk3d0 did not complete successfully Expected <string>: map[apiVersion:sparkoperator.k8s.io/v1beta2 kind:SparkApplication metadata:map[creationTimestamp:2026-09-15T15:23:51Z generation:1 managedFields:[map[apiVersion:sparkoperator.k8s.io/v1beta2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[.:map[] f:arguments:map[] f:driver:map[.:map[] f:coreLimit:map[] f:cores:map[] f:labels:map[.:map[] f:version:map[]] f:memory:map[] f:securityContext:map[] f:serviceAccount:map[] f:volumeMounts:map[]] f:executor:map[.:map[] f:cores:map[] f:instances:map[] f:labels:map[.:map[] f:version:map[]] f:memory:map[] f:securityContext:map[] f:volumeMounts:map[]] f:image:map[] f:imagePullPolicy:map[] f:mainApplicationFile:map[] f:mainClass:map[] f:mode:map[] f:restartPolicy:map[.:map[] f:type:map[]] f:sparkConf:map[.:map[] f:spark.blockManager.port:map[] f:spark.driver.blockManager.port:map[] f:spark.driver.port:map[] f:spark.port.maxRetries:map[]] f:sparkVersion:map[] f:type:map[] f:volumes:map[]]] manager:e2e-tests operation:Update time:2026-09-15T15:23:51Z] map[apiVersion:sparkoperator.k8s.io/v1beta2 fieldsType:FieldsV1 fieldsV1:map[f:status:map[.:map[] f:applicationState:map[.:map[] f:state:map[]] f:driverInfo:map[.:map[] f:podName:map[] f:webUIAddress:map[] f:webUIPort:map[] f:webUIServiceName:map[]] f:executionAttempts:map[] f:executorState:map[.:map[] f:spark-pi-1c6035a0a5ab7487-exec-1:map[]] f:lastSubmissionAttemptTime:map[] f:sparkApplicationId:map[] f:submissionAttempts:map[] f:submissionID:map[] f:terminationTime:map[]]] manager:spark-operator operation:Update subresource:status time:2026-09-15T15:24:35Z]] name:spark-pi-dakm61v7fmuc7nkvk3d0 namespace:redhat-ods-applications resourceVersion:129614 uid:23a127cf-48b3-47e5-9699-03f49ec5fcf3] spec:map[arguments:[1000] deps:map[] driver:map[coreLimit:1200m cores:1 labels:map[version:4.0.1] memory:512m securityContext:map[] serviceAccount:spark-operator-spark volumeMounts:[map[mountPath:/opt/spark/work-dir name:spark-work-dir]]] executor:map[cores:1 instances:1 labels:map[version:4.0.1] memory:512m securityContext:map[] volumeMounts:[map[mountPath:/opt/spark/work-dir name:spark-work-dir]]] image:quay.io/opendatahub/data-processing:Spark-v4.0.1 imagePullPolicy:IfNotPresent mainApplicationFile:local:///opt/spark/examples/jars/spark-examples_2.13-4.0.1.jar mainClass:org.apache.spark.examples.SparkPi mode:cluster restartPolicy:map[type:Never] sparkConf:map[spark.blockManager.port:8081 spark.driver.blockManager.port:8082 spark.driver.port:8080 spark.port.maxRetries:0] sparkVersion:4.0.1 type:Scala volumes:[map[emptyDir:map[] name:spark-work-dir]]] status:map[applicationState:map[state:RUNNING] driverInfo:map[podName:spark-pi-dakm61v7fmuc7nkvk3d0-driver webUIAddress:172.31.183.110:4040 webUIPort:4040 webUIServiceName:spark-pi-dakm61v7fmuc7nkvk3d0-ui-svc] executionAttempts:1 executorState:map[spark-pi-1c6035a0a5ab7487-exec-1:PENDING] lastSubmissionAttemptTime:2026-09-15T15:23:51Z sparkApplicationId:spark-88c9721b9051483cbf3103505d77d703 submissionAttempts:1 submissionID:78009c7e-b7a2-4eef-b5e0-7888bbc4394f terminationTime:<nil>]] to match expression <string>: .status.applicationState.state == "COMPLETED" --- FAIL: TestOdhOperator/components/group_1/sparkoperator (371.31s) === RUN TestOdhOperator/components/group_1 controller_test.go:753: Skipping tests for components/aigateway controller_test.go:753: Skipping tests for components/dashboard controller_test.go:753: Skipping tests for components/datasciencepipelines controller_test.go:753: Skipping tests for components/feastoperator controller_test.go:753: Skipping tests for components/kserve controller_test.go:753: Skipping tests for components/modelregistry controller_test.go:753: Skipping tests for components/ogx controller_test.go:753: Skipping tests for components/ray controller_test.go:753: Skipping tests for components/trainer controller_test.go:753: Skipping tests for components/trainingoperator controller_test.go:753: Skipping tests for components/workbenches --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components 2026/09/15 15:28:51 TEST FAILURE DETECTED: group 1 2026/09/15 15:28:51 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/09/15 15:28:52 === CLUSTER STATE === 2026/09/15 15:28:52 Node: ip-10-0-130-52.ec2.internal 2026/09/15 15:28:52 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/15 15:28:52 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/15 15:28:52 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/15 15:28:52 Condition Ready: True - kubelet is posting ready status 2026/09/15 15:28:52 Resources: 7500m CPU, 31236216Ki memory allocatable (of 8000m CPU, 32387192Ki memory capacity) 2026/09/15 15:28:52 Node: ip-10-0-137-99.ec2.internal 2026/09/15 15:28:52 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/15 15:28:52 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/15 15:28:52 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/15 15:28:52 Condition Ready: True - kubelet is posting ready status 2026/09/15 15:28:52 Resources: 7500m CPU, 31236212Ki memory allocatable (of 8000m CPU, 32387188Ki memory capacity) 2026/09/15 15:28:52 Node: ip-10-0-141-236.ec2.internal 2026/09/15 15:28:52 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/15 15:28:52 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/15 15:28:52 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/15 15:28:52 Condition Ready: True - kubelet is posting ready status 2026/09/15 15:28:52 Resources: 7500m CPU, 31236208Ki memory allocatable (of 8000m CPU, 32387184Ki memory capacity) 2026/09/15 15:28:52 === DEPLOYMENTS === 2026/09/15 15:28:52 Namespace: redhat-ods-applications 2026/09/15 15:28:52 Deployment dashboard-redirect: 2/2 (READY) 2026/09/15 15:28:52 Deployment data-science-pipelines-operator-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment kserve-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment kuberay-operator: 1/1 (READY) 2026/09/15 15:28:52 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment maas-api: 1/1 (READY) 2026/09/15 15:28:52 Deployment maas-controller: 1/1 (READY) 2026/09/15 15:28:52 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment model-serving-api: 1/1 (READY) 2026/09/15 15:28:52 Deployment notebook-controller-deployment: 1/1 (READY) 2026/09/15 15:28:52 Deployment odh-model-controller: 1/1 (READY) 2026/09/15 15:28:52 Deployment odh-notebook-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment ogx-k8s-operator-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment rhods-dashboard: 2/2 (READY) 2026/09/15 15:28:52 Deployment spark-operator-controller: 2/2 (READY) 2026/09/15 15:28:52 Deployment spark-operator-webhook: 2/2 (READY) 2026/09/15 15:28:52 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Namespace: kube-system 2026/09/15 15:28:52 === PODS === 2026/09/15 15:28:52 Namespace: redhat-ods-applications 2026/09/15 15:28:52 Pod maas-api-key-cleanup-29824725-mtj4q: Succeeded 2026/09/15 15:28:52 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/09/15 15:28:52 Pod maas-api-key-cleanup-29824740-n5j46: Succeeded 2026/09/15 15:28:52 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/09/15 15:28:52 Pod maas-api-key-cleanup-29824755-mvc29: Succeeded 2026/09/15 15:28:52 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/09/15 15:28:52 Namespace kube-system: no problematic pods found 2026/09/15 15:28:52 === LOGS (CURRENT) for container cleanup === 2026/09/15 15:28:52 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/09/15 15:28:52 === END LOGS === 2026/09/15 15:28:52 === LOGS (CURRENT) for container cleanup === 2026/09/15 15:28:52 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/09/15 15:28:52 === END LOGS === 2026/09/15 15:28:52 === LOGS (CURRENT) for container cleanup === 2026/09/15 15:28:52 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/09/15 15:28:52 === END LOGS === 2026/09/15 15:28:52 === OPERATOR STATUS === 2026/09/15 15:28:52 Operator deployment: 3/3 ready 2026/09/15 15:28:52 === DSCI STATUS === 2026/09/15 15:28:52 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/09/15 15:28:52 === DSC STATUS === 2026/09/15 15:28:52 DSC default-dsc: 2026/09/15 15:28:52 Ready: True - 2026/09/15 15:28:52 ProvisioningSucceeded: True - 2026/09/15 15:28:52 ComponentsReady: True - 2026/09/15 15:28:52 AIPipelinesReady: True - 2026/09/15 15:28:52 DashboardReady: True - 2026/09/15 15:28:52 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/09/15 15:28:52 KserveLLMInferenceServiceDependencies: True - 2026/09/15 15:28:52 KserveLLMInferenceServiceWideEPDependencies: True - 2026/09/15 15:28:52 KserveReady: True - 2026/09/15 15:28:52 KueueReady: False - Component ManagementState is set to Removed 2026/09/15 15:28:52 LLM-D-WVADependencies: True - 2026/09/15 15:28:52 MLflowOperatorReady: True - 2026/09/15 15:28:52 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/09/15 15:28:52 ModelControllerReady: True - 2026/09/15 15:28:52 ModelRegistryReady: True - 2026/09/15 15:28:52 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/09/15 15:28:52 OGXReady: True - 2026/09/15 15:28:52 RayReady: True - 2026/09/15 15:28:52 SparkOperatorReady: True - 2026/09/15 15:28:52 TrainerReady: True - 2026/09/15 15:28:52 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/09/15 15:28:52 TrustyAIReady: True - 2026/09/15 15:28:52 WorkbenchesReady: True - 2026/09/15 15:28:52 === RECENT EVENTS === 2026/09/15 15:28:52 15:28:51 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: Killing - Stopping container spark-kubernetes-driver 2026/09/15 15:28:52 15:24:35 WARN Pod/spark-pi-1c6035a0a5ab7487-exec-1: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/09/15 15:28:52 15:24:35 INFO SparkApplication/spark-pi-dakm61v7fmuc7nkvk3d0: SparkExecutorPending - Executor spark-pi-1c6035a0a5ab7487-exec-1 is pending 2026/09/15 15:28:52 15:24:27 INFO SparkApplication/spark-pi-dakm61v7fmuc7nkvk3d0: SparkDriverRunning - Driver spark-pi-dakm61v7fmuc7nkvk3d0-driver is running 2026/09/15 15:28:52 15:24:26 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: Pulled - Successfully pulled image "quay.io/opendatahub/data-processing:Spark-v4.0.1" in 17.646s (17.646s including waiting). Image size: 1778928657 bytes. 2026/09/15 15:28:52 15:24:26 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: Created - Created container: spark-kubernetes-driver 2026/09/15 15:28:52 15:24:26 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: Started - Started container spark-kubernetes-driver 2026/09/15 15:28:52 15:24:08 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: Pulling - Pulling image "quay.io/opendatahub/data-processing:Spark-v4.0.1" 2026/09/15 15:28:52 15:24:08 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: AddedInterface - Add eth0 [10.132.0.120/23] from ovn-kubernetes 2026/09/15 15:28:52 15:24:08 INFO SparkApplication/spark-pi-dakm61v7fmuc7nkvk3d0: SparkApplicationSubmitted - SparkApplication spark-pi-dakm61v7fmuc7nkvk3d0 was submitted successfully 2026/09/15 15:28:52 15:24:07 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-dakm61v7fmuc7nkvk3d0-driver to ip-10-0-130-52.ec2.internal 2026/09/15 15:28:52 15:24:07 WARN Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-dakm61v7fmuc7nkvk3d0-14ad37a0a5aaf7c0-driver-podspec-conf-map" not found 2026/09/15 15:28:52 15:24:07 WARN Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-7fa46fa0a5ab0d3f-conf-map" not found 2026/09/15 15:28:52 === RESOURCE QUOTAS === 2026/09/15 15:28:52 No resource quotas found in monitored namespaces 2026/09/15 15:28:52 Classification for group 1: infrastructure/storage (code=1006, confidence=medium) 2026/09/15 15:28:52 Evidence: event Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-dakm61v7fmuc7nkvk3d0-14ad37a0a5aaf7c0-driver-podspec-conf-map" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: FailedMount - MountVolume.SetUp failed for volume \"pod-template-volume\" : configmap \"spark-pi-dakm61v7fmuc7nkvk3d0-14ad37a0a5aaf7c0-driver-podspec-conf-map\" not found"],"confidence":"medium"} 2026/09/15 15:28:52 Diagnostics complete for failed test: group 1 controller_test.go:753: Stopping: group 1 test failed. 2026/09/15 15:28:52 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/09/15 15:28:52 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/09/15 15:28:52 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (371.91s) === RUN TestOdhOperator 2026/09/15 15:22:29 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/09/15 15:28:52 TEST FAILURE DETECTED: components 2026/09/15 15:28:52 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/09/15 15:28:52 === CLUSTER STATE === 2026/09/15 15:28:52 Node: ip-10-0-130-52.ec2.internal 2026/09/15 15:28:52 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/15 15:28:52 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/15 15:28:52 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/15 15:28:52 Condition Ready: True - kubelet is posting ready status 2026/09/15 15:28:52 Resources: 7500m CPU, 31236216Ki memory allocatable (of 8000m CPU, 32387192Ki memory capacity) 2026/09/15 15:28:52 Node: ip-10-0-137-99.ec2.internal 2026/09/15 15:28:52 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/15 15:28:52 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/15 15:28:52 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/15 15:28:52 Condition Ready: True - kubelet is posting ready status 2026/09/15 15:28:52 Resources: 7500m CPU, 31236212Ki memory allocatable (of 8000m CPU, 32387188Ki memory capacity) 2026/09/15 15:28:52 Node: ip-10-0-141-236.ec2.internal 2026/09/15 15:28:52 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/15 15:28:52 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/15 15:28:52 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/15 15:28:52 Condition Ready: True - kubelet is posting ready status 2026/09/15 15:28:52 Resources: 7500m CPU, 31236208Ki memory allocatable (of 8000m CPU, 32387184Ki memory capacity) 2026/09/15 15:28:52 === DEPLOYMENTS === 2026/09/15 15:28:52 Namespace: kube-system 2026/09/15 15:28:52 Namespace: redhat-ods-applications 2026/09/15 15:28:52 Deployment dashboard-redirect: 2/2 (READY) 2026/09/15 15:28:52 Deployment data-science-pipelines-operator-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment kserve-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment kuberay-operator: 1/1 (READY) 2026/09/15 15:28:52 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment maas-api: 1/1 (READY) 2026/09/15 15:28:52 Deployment maas-controller: 1/1 (READY) 2026/09/15 15:28:52 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment model-serving-api: 1/1 (READY) 2026/09/15 15:28:52 Deployment notebook-controller-deployment: 1/1 (READY) 2026/09/15 15:28:52 Deployment odh-model-controller: 1/1 (READY) 2026/09/15 15:28:52 Deployment odh-notebook-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment ogx-k8s-operator-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 Deployment rhods-dashboard: 2/2 (READY) 2026/09/15 15:28:52 Deployment spark-operator-controller: 2/2 (READY) 2026/09/15 15:28:52 Deployment spark-operator-webhook: 2/2 (READY) 2026/09/15 15:28:52 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/09/15 15:28:52 === PODS === 2026/09/15 15:28:52 Failed to collect pod data: redhat-ods-applications/spark-pi-dakm61v7fmuc7nkvk3d0-driver: phase=Failed 2026/09/15 15:28:52 === OPERATOR STATUS === 2026/09/15 15:28:52 Operator deployment: 3/3 ready 2026/09/15 15:28:52 === DSCI STATUS === 2026/09/15 15:28:52 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/09/15 15:28:52 === DSC STATUS === 2026/09/15 15:28:52 DSC default-dsc: 2026/09/15 15:28:52 Ready: True - 2026/09/15 15:28:52 ProvisioningSucceeded: True - 2026/09/15 15:28:52 ComponentsReady: True - 2026/09/15 15:28:52 AIPipelinesReady: True - 2026/09/15 15:28:52 DashboardReady: True - 2026/09/15 15:28:52 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/09/15 15:28:52 KserveLLMInferenceServiceDependencies: True - 2026/09/15 15:28:52 KserveLLMInferenceServiceWideEPDependencies: True - 2026/09/15 15:28:52 KserveReady: True - 2026/09/15 15:28:52 KueueReady: False - Component ManagementState is set to Removed 2026/09/15 15:28:52 LLM-D-WVADependencies: True - 2026/09/15 15:28:52 MLflowOperatorReady: True - 2026/09/15 15:28:52 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/09/15 15:28:52 ModelControllerReady: True - 2026/09/15 15:28:52 ModelRegistryReady: True - 2026/09/15 15:28:52 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/09/15 15:28:52 OGXReady: True - 2026/09/15 15:28:52 RayReady: True - 2026/09/15 15:28:52 SparkOperatorReady: True - 2026/09/15 15:28:52 TrainerReady: True - 2026/09/15 15:28:52 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/09/15 15:28:52 TrustyAIReady: True - 2026/09/15 15:28:52 WorkbenchesReady: True - 2026/09/15 15:28:52 === RECENT EVENTS === 2026/09/15 15:28:52 15:28:51 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: Killing - Stopping container spark-kubernetes-driver 2026/09/15 15:28:52 15:24:35 WARN Pod/spark-pi-1c6035a0a5ab7487-exec-1: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/09/15 15:28:52 15:24:35 INFO SparkApplication/spark-pi-dakm61v7fmuc7nkvk3d0: SparkExecutorPending - Executor spark-pi-1c6035a0a5ab7487-exec-1 is pending 2026/09/15 15:28:52 15:24:27 INFO SparkApplication/spark-pi-dakm61v7fmuc7nkvk3d0: SparkDriverRunning - Driver spark-pi-dakm61v7fmuc7nkvk3d0-driver is running 2026/09/15 15:28:52 15:24:26 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: Pulled - Successfully pulled image "quay.io/opendatahub/data-processing:Spark-v4.0.1" in 17.646s (17.646s including waiting). Image size: 1778928657 bytes. 2026/09/15 15:28:52 15:24:26 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: Created - Created container: spark-kubernetes-driver 2026/09/15 15:28:52 15:24:26 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: Started - Started container spark-kubernetes-driver 2026/09/15 15:28:52 15:24:08 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: Pulling - Pulling image "quay.io/opendatahub/data-processing:Spark-v4.0.1" 2026/09/15 15:28:52 15:24:08 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: AddedInterface - Add eth0 [10.132.0.120/23] from ovn-kubernetes 2026/09/15 15:28:52 15:24:08 INFO SparkApplication/spark-pi-dakm61v7fmuc7nkvk3d0: SparkApplicationSubmitted - SparkApplication spark-pi-dakm61v7fmuc7nkvk3d0 was submitted successfully 2026/09/15 15:28:52 15:24:07 INFO Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-dakm61v7fmuc7nkvk3d0-driver to ip-10-0-130-52.ec2.internal 2026/09/15 15:28:52 15:24:07 WARN Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-dakm61v7fmuc7nkvk3d0-14ad37a0a5aaf7c0-driver-podspec-conf-map" not found 2026/09/15 15:28:52 15:24:07 WARN Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-7fa46fa0a5ab0d3f-conf-map" not found 2026/09/15 15:28:52 === RESOURCE QUOTAS === 2026/09/15 15:28:52 No resource quotas found in monitored namespaces 2026/09/15 15:28:52 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/09/15 15:28:52 Evidence: event Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-dakm61v7fmuc7nkvk3d0-14ad37a0a5aaf7c0-driver-podspec-conf-map" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/spark-pi-dakm61v7fmuc7nkvk3d0-driver: FailedMount - MountVolume.SetUp failed for volume \"pod-template-volume\" : configmap \"spark-pi-dakm61v7fmuc7nkvk3d0-14ad37a0a5aaf7c0-driver-podspec-conf-map\" not found"],"confidence":"medium"} 2026/09/15 15:28:52 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/09/15 15:28:52 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/09/15 15:28:52 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/09/15 15:28:53 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (384.29s)