=== RUN TestOdhOperator/components/group_1/sparkoperator/Validate_resource_deletion_recovery/ConfigMap_deletion_recovery/ConfigMap_spark-operator-params-mmh2bg9h45 testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/sparkoperator/Validate_resource_deletion_recovery/ConfigMap_deletion_recovery/ConfigMap_spark-operator-params-mmh2bg9h45 (95.04s) 2026/07/02 07:42:57 TEST FAILURE DETECTED: ConfigMap deletion recovery 2026/07/02 07:42:57 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:ConfigMap deletion recovery) === 2026/07/02 07:42:57 clusterhealth: node metrics unavailable (Metrics Server may not be installed): the server is currently unable to handle the request (get nodes.metrics.k8s.io) 2026/07/02 07:42:58 === CLUSTER STATE === 2026/07/02 07:42:58 Failed to collect node data: unhealthy nodes: ip-10-0-136-131.ec2.internal (Ready=Unknown) 2026/07/02 07:42:58 === DEPLOYMENTS === 2026/07/02 07:42:58 Failed to collect deployment data: deployments not ready: redhat-ods-applications/postgres (0/1), redhat-ods-applications/spark-operator-controller (0/2), redhat-ods-applications/spark-operator-webhook (0/2) 2026/07/02 07:42:58 === PODS === 2026/07/02 07:42:58 Namespace redhat-ods-applications: no problematic pods found 2026/07/02 07:42:58 Namespace kube-system: no problematic pods found 2026/07/02 07:42:58 === OPERATOR STATUS === 2026/07/02 07:42:58 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; dependent kueue: deployment 0/2 ready; dependent jobset: deployment 0/1 ready; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; dependent kuadrant: deployment 0/1 ready 2026/07/02 07:42:58 === DSCI STATUS === 2026/07/02 07:42:58 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/02 07:42:58 === DSC STATUS === 2026/07/02 07:42:58 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/02 07:42:58 === RECENT EVENTS === 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-njl6n 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-controller-5dd7d48f87-t2zf8: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-t2zf8 2026/07/02 07:42:58 07:42:17 WARN Pod/postgres-69fb4594cf-vlkn9: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:17 WARN Pod/konnectivity-agent-b6bsq: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:17 WARN Pod/global-pull-secret-syncer-569jc: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-webhook-5f8768dbfd-ddfp2: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-ddfp2 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-fhxd9 2026/07/02 07:42:58 07:42:16 WARN Pod/kube-apiserver-proxy-ip-10-0-130-228.ec2.internal: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:10 WARN Pod/spark-operator-controller-5dd7d48f87-t2zf8: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:10 WARN Pod/spark-operator-webhook-5f8768dbfd-fhxd9: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/global-pull-secret-syncer-ldz9z: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/spark-operator-webhook-5f8768dbfd-ddfp2: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/kube-apiserver-proxy-ip-10-0-134-81.ec2.internal: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/konnectivity-agent-mqjqx: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:03 WARN Pod/global-pull-secret-syncer-zrkbn: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:03 WARN Pod/spark-operator-controller-5dd7d48f87-njl6n: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:02 WARN Pod/konnectivity-agent-44png: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:02 WARN Pod/kube-apiserver-proxy-ip-10-0-136-131.ec2.internal: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:41:14 INFO Lease/cert-manager-controller: LeaderElection - cert-manager-59c77684cc-v45j9-external-cert-manager-controller stopped leading 2026/07/02 07:42:58 07:41:13 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkDriverCompleted - Driver scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver completed 2026/07/02 07:42:58 07:41:11 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkExecutorCompleted - Executor spark-pi-26d30b9f21c60382-exec-1 completed 2026/07/02 07:42:58 07:41:11 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/02 07:42:58 07:40:57 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkExecutorRunning - Executor spark-pi-26d30b9f21c60382-exec-1 is running 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-26d30b9f21c60382-exec-1 to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:40:56 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkExecutorPending - Executor spark-pi-26d30b9f21c60382-exec-1 is pending 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Started - Started container spark-kubernetes-executor 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: AddedInterface - Add eth0 [10.134.0.83/23] from ovn-kubernetes 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/02 07:42:58 07:40:47 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkDriverRunning - Driver scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver is running 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Started - Started container spark-kubernetes-driver 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Created - Created container: spark-kubernetes-driver 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: AddedInterface - Add eth0 [10.134.0.82/23] from ovn-kubernetes 2026/07/02 07:42:58 07:40:46 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Scheduled - Successfully assigned redhat-ods-applications/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:40:46 WARN Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-6395959f21c5e0d4-conf-map" not found 2026/07/02 07:42:58 07:40:46 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkApplicationSubmitted - SparkApplication scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717 was submitted successfully 2026/07/02 07:42:58 07:40:46 WARN Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-ad90a89f21c5ce81-driver-podspec-conf-map" not found 2026/07/02 07:42:58 07:39:56 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkDriverCompleted - Driver spark-pi-d931b31sidhc74ll6br0-driver completed 2026/07/02 07:42:58 07:39:53 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/02 07:42:58 07:39:53 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkExecutorCompleted - Executor spark-pi-c449739f21c4d30b-exec-1 completed 2026/07/02 07:42:58 07:39:39 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkExecutorRunning - Executor spark-pi-c449739f21c4d30b-exec-1 is running 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Started - Started container spark-kubernetes-executor 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: AddedInterface - Add eth0 [10.134.0.81/23] from ovn-kubernetes 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/02 07:42:58 07:39:37 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-c449739f21c4d30b-exec-1 to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:39:37 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkExecutorPending - Executor spark-pi-c449739f21c4d30b-exec-1 is pending 2026/07/02 07:42:58 07:39:30 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkDriverRunning - Driver spark-pi-d931b31sidhc74ll6br0-driver is running 2026/07/02 07:42:58 07:39:29 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Pulled - Successfully pulled image "quay.io/opendatahub/data-processing:Spark-v4.0.1" in 22.419s (22.419s including waiting). Image size: 1778928657 bytes. 2026/07/02 07:42:58 07:39:29 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Started - Started container spark-kubernetes-driver 2026/07/02 07:42:58 07:39:29 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Created - Created container: spark-kubernetes-driver 2026/07/02 07:42:58 07:39:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Created - Created container: controller 2026/07/02 07:42:58 07:39:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Pulled - Successfully pulled image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" in 1m4.343s (1m4.343s including waiting). Image size: 2736477781 bytes. 2026/07/02 07:42:58 07:39:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Started - Started container controller 2026/07/02 07:42:58 07:39:06 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-d931b31sidhc74ll6br0-driver to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:39:06 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Pulling - Pulling image "quay.io/opendatahub/data-processing:Spark-v4.0.1" 2026/07/02 07:42:58 07:39:06 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: AddedInterface - Add eth0 [10.134.0.80/23] from ovn-kubernetes 2026/07/02 07:42:58 07:39:06 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkApplicationSubmitted - SparkApplication spark-pi-d931b31sidhc74ll6br0 was submitted successfully 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-fhxd9 to ip-10-0-134-81.ec2.internal 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: AddedInterface - Add eth0 [10.133.0.151/23] from ovn-kubernetes 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Created - Created container: webhook 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Started - Started container webhook 2026/07/02 07:42:58 07:38:32 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-fhxd9 2026/07/02 07:42:58 07:38:32 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 1 to 2 2026/07/02 07:42:58 07:38:13 INFO Lease/spark-operator-webhook-lock: LeaderElection - spark-operator-webhook-5f8768dbfd-ddfp2_4b5d8c5e-db00-4781-ba61-6abd8706c068 became leader 2026/07/02 07:42:58 07:38:13 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Pulling - Pulling image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" 2026/07/02 07:42:58 07:38:13 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: AddedInterface - Add eth0 [10.134.0.79/23] from ovn-kubernetes 2026/07/02 07:42:58 07:38:12 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-njl6n to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:38:12 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 1 to 2 2026/07/02 07:42:58 07:38:12 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-njl6n 2026/07/02 07:42:58 07:38:11 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-5dd7d48f87-t2zf8_87362461-b81e-4570-8a17-e0a190c38fc9 became leader 2026/07/02 07:42:58 === RESOURCE QUOTAS === 2026/07/02 07:42:58 No resource quotas found in monitored namespaces 2026/07/02 07:42:58 Classification for ConfigMap deletion recovery: infrastructure/storage (code=1006, confidence=medium) 2026/07/02 07:42:58 Evidence: event Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-6395959f21c5e0d4-conf-map" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume \"spark-conf-volume-driver\" : configmap \"spark-drv-6395959f21c5e0d4-conf-map\" not found"],"confidence":"medium"} 2026/07/02 07:42:58 Diagnostics complete for failed test: ConfigMap deletion recovery === RUN TestOdhOperator/components/group_1/sparkoperator/Validate_resource_deletion_recovery/ConfigMap_deletion_recovery testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/sparkoperator/Validate_resource_deletion_recovery/ConfigMap_deletion_recovery (95.24s) 2026/07/02 07:42:58 TEST FAILURE DETECTED: Validate resource deletion recovery 2026/07/02 07:42:58 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:Validate resource deletion recovery) === 2026/07/02 07:42:58 clusterhealth: node metrics unavailable (Metrics Server may not be installed): the server is currently unable to handle the request (get nodes.metrics.k8s.io) 2026/07/02 07:42:58 === CLUSTER STATE === 2026/07/02 07:42:58 Failed to collect node data: unhealthy nodes: ip-10-0-136-131.ec2.internal (Ready=Unknown) 2026/07/02 07:42:58 === DEPLOYMENTS === 2026/07/02 07:42:58 Failed to collect deployment data: deployments not ready: redhat-ods-applications/postgres (0/1), redhat-ods-applications/spark-operator-controller (0/2), redhat-ods-applications/spark-operator-webhook (0/2) 2026/07/02 07:42:58 === PODS === 2026/07/02 07:42:58 Namespace redhat-ods-applications: no problematic pods found 2026/07/02 07:42:58 Namespace kube-system: no problematic pods found 2026/07/02 07:42:58 === OPERATOR STATUS === 2026/07/02 07:42:58 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; dependent kueue: deployment 0/2 ready; dependent jobset: deployment 0/1 ready; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; dependent kuadrant: deployment 0/1 ready 2026/07/02 07:42:58 === DSCI STATUS === 2026/07/02 07:42:58 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/02 07:42:58 === DSC STATUS === 2026/07/02 07:42:58 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/02 07:42:58 === RECENT EVENTS === 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-njl6n 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-controller-5dd7d48f87-t2zf8: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-t2zf8 2026/07/02 07:42:58 07:42:17 WARN Pod/postgres-69fb4594cf-vlkn9: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:17 WARN Pod/konnectivity-agent-b6bsq: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:17 WARN Pod/global-pull-secret-syncer-569jc: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-webhook-5f8768dbfd-ddfp2: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-ddfp2 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-fhxd9 2026/07/02 07:42:58 07:42:16 WARN Pod/kube-apiserver-proxy-ip-10-0-130-228.ec2.internal: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:10 WARN Pod/spark-operator-controller-5dd7d48f87-t2zf8: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:10 WARN Pod/spark-operator-webhook-5f8768dbfd-fhxd9: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/global-pull-secret-syncer-ldz9z: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/spark-operator-webhook-5f8768dbfd-ddfp2: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/kube-apiserver-proxy-ip-10-0-134-81.ec2.internal: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/konnectivity-agent-mqjqx: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:03 WARN Pod/global-pull-secret-syncer-zrkbn: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:03 WARN Pod/spark-operator-controller-5dd7d48f87-njl6n: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:02 WARN Pod/konnectivity-agent-44png: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:02 WARN Pod/kube-apiserver-proxy-ip-10-0-136-131.ec2.internal: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:41:14 INFO Lease/cert-manager-controller: LeaderElection - cert-manager-59c77684cc-v45j9-external-cert-manager-controller stopped leading 2026/07/02 07:42:58 07:41:13 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkDriverCompleted - Driver scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver completed 2026/07/02 07:42:58 07:41:11 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkExecutorCompleted - Executor spark-pi-26d30b9f21c60382-exec-1 completed 2026/07/02 07:42:58 07:41:11 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/02 07:42:58 07:40:57 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkExecutorRunning - Executor spark-pi-26d30b9f21c60382-exec-1 is running 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-26d30b9f21c60382-exec-1 to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:40:56 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkExecutorPending - Executor spark-pi-26d30b9f21c60382-exec-1 is pending 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Started - Started container spark-kubernetes-executor 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: AddedInterface - Add eth0 [10.134.0.83/23] from ovn-kubernetes 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/02 07:42:58 07:40:47 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkDriverRunning - Driver scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver is running 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Started - Started container spark-kubernetes-driver 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Created - Created container: spark-kubernetes-driver 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: AddedInterface - Add eth0 [10.134.0.82/23] from ovn-kubernetes 2026/07/02 07:42:58 07:40:46 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Scheduled - Successfully assigned redhat-ods-applications/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:40:46 WARN Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-6395959f21c5e0d4-conf-map" not found 2026/07/02 07:42:58 07:40:46 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkApplicationSubmitted - SparkApplication scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717 was submitted successfully 2026/07/02 07:42:58 07:40:46 WARN Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-ad90a89f21c5ce81-driver-podspec-conf-map" not found 2026/07/02 07:42:58 07:39:56 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkDriverCompleted - Driver spark-pi-d931b31sidhc74ll6br0-driver completed 2026/07/02 07:42:58 07:39:53 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/02 07:42:58 07:39:53 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkExecutorCompleted - Executor spark-pi-c449739f21c4d30b-exec-1 completed 2026/07/02 07:42:58 07:39:39 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkExecutorRunning - Executor spark-pi-c449739f21c4d30b-exec-1 is running 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Started - Started container spark-kubernetes-executor 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: AddedInterface - Add eth0 [10.134.0.81/23] from ovn-kubernetes 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/02 07:42:58 07:39:37 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-c449739f21c4d30b-exec-1 to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:39:37 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkExecutorPending - Executor spark-pi-c449739f21c4d30b-exec-1 is pending 2026/07/02 07:42:58 07:39:30 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkDriverRunning - Driver spark-pi-d931b31sidhc74ll6br0-driver is running 2026/07/02 07:42:58 07:39:29 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Pulled - Successfully pulled image "quay.io/opendatahub/data-processing:Spark-v4.0.1" in 22.419s (22.419s including waiting). Image size: 1778928657 bytes. 2026/07/02 07:42:58 07:39:29 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Started - Started container spark-kubernetes-driver 2026/07/02 07:42:58 07:39:29 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Created - Created container: spark-kubernetes-driver 2026/07/02 07:42:58 07:39:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Created - Created container: controller 2026/07/02 07:42:58 07:39:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Pulled - Successfully pulled image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" in 1m4.343s (1m4.343s including waiting). Image size: 2736477781 bytes. 2026/07/02 07:42:58 07:39:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Started - Started container controller 2026/07/02 07:42:58 07:39:06 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-d931b31sidhc74ll6br0-driver to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:39:06 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Pulling - Pulling image "quay.io/opendatahub/data-processing:Spark-v4.0.1" 2026/07/02 07:42:58 07:39:06 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: AddedInterface - Add eth0 [10.134.0.80/23] from ovn-kubernetes 2026/07/02 07:42:58 07:39:06 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkApplicationSubmitted - SparkApplication spark-pi-d931b31sidhc74ll6br0 was submitted successfully 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-fhxd9 to ip-10-0-134-81.ec2.internal 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: AddedInterface - Add eth0 [10.133.0.151/23] from ovn-kubernetes 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Created - Created container: webhook 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Started - Started container webhook 2026/07/02 07:42:58 07:38:32 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-fhxd9 2026/07/02 07:42:58 07:38:32 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 1 to 2 2026/07/02 07:42:58 07:38:13 INFO Lease/spark-operator-webhook-lock: LeaderElection - spark-operator-webhook-5f8768dbfd-ddfp2_4b5d8c5e-db00-4781-ba61-6abd8706c068 became leader 2026/07/02 07:42:58 07:38:13 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Pulling - Pulling image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" 2026/07/02 07:42:58 07:38:13 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: AddedInterface - Add eth0 [10.134.0.79/23] from ovn-kubernetes 2026/07/02 07:42:58 07:38:12 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-njl6n to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:38:12 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 1 to 2 2026/07/02 07:42:58 07:38:12 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-njl6n 2026/07/02 07:42:58 07:38:11 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-5dd7d48f87-t2zf8_87362461-b81e-4570-8a17-e0a190c38fc9 became leader 2026/07/02 07:42:58 === RESOURCE QUOTAS === 2026/07/02 07:42:58 No resource quotas found in monitored namespaces 2026/07/02 07:42:58 Classification for Validate resource deletion recovery: infrastructure/storage (code=1006, confidence=medium) 2026/07/02 07:42:58 Evidence: event Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-6395959f21c5e0d4-conf-map" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume \"spark-conf-volume-driver\" : configmap \"spark-drv-6395959f21c5e0d4-conf-map\" not found"],"confidence":"medium"} 2026/07/02 07:42:58 Diagnostics complete for failed test: Validate resource deletion recovery === RUN TestOdhOperator/components/group_1/sparkoperator/Validate_resource_deletion_recovery testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/sparkoperator/Validate_resource_deletion_recovery (95.41s) === RUN TestOdhOperator/components/group_1/sparkoperator === PAUSE TestOdhOperator/components/group_1/sparkoperator === CONT TestOdhOperator/components/group_1/sparkoperator 2026-07-02T07:37:51Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.22"}} test_context_test.go:1321: Timed out after 90.001s. Resource ConfigMap redhat-ods-applications/spark-operator-params-mmh2bg9h45 was not properly recreated with new identity The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1302 with: Expected ConfigMap redhat-ods-applications/spark-operator-params-mmh2bg9h45 to be recreated Expected <*unstructured.Unstructured | 0x0>: nil not to be nil --- FAIL: TestOdhOperator/components/group_1/sparkoperator (306.33s) === RUN TestOdhOperator/components/group_1 controller_test.go:763: Skipping tests for components/aigateway controller_test.go:763: Skipping tests for components/dashboard controller_test.go:763: Skipping tests for components/datasciencepipelines controller_test.go:763: Skipping tests for components/feastoperator controller_test.go:763: Skipping tests for components/kserve controller_test.go:763: Skipping tests for components/modelregistry controller_test.go:763: Skipping tests for components/ogx controller_test.go:763: Skipping tests for components/ray controller_test.go:763: Skipping tests for components/trainer controller_test.go:763: Skipping tests for components/trainingoperator controller_test.go:763: Skipping tests for components/workbenches --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components 2026/07/02 07:42:58 TEST FAILURE DETECTED: group 1 2026/07/02 07:42:58 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/02 07:42:58 clusterhealth: node metrics unavailable (Metrics Server may not be installed): the server is currently unable to handle the request (get nodes.metrics.k8s.io) 2026/07/02 07:42:58 === CLUSTER STATE === 2026/07/02 07:42:58 Failed to collect node data: unhealthy nodes: ip-10-0-136-131.ec2.internal (Ready=Unknown) 2026/07/02 07:42:58 === DEPLOYMENTS === 2026/07/02 07:42:58 Failed to collect deployment data: deployments not ready: redhat-ods-applications/postgres (0/1), redhat-ods-applications/spark-operator-controller (0/2), redhat-ods-applications/spark-operator-webhook (0/2) 2026/07/02 07:42:58 === PODS === 2026/07/02 07:42:58 Namespace redhat-ods-applications: no problematic pods found 2026/07/02 07:42:58 Namespace kube-system: no problematic pods found 2026/07/02 07:42:58 === OPERATOR STATUS === 2026/07/02 07:42:58 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; dependent kueue: deployment 0/2 ready; dependent jobset: deployment 0/1 ready; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; dependent kuadrant: deployment 0/1 ready 2026/07/02 07:42:58 === DSCI STATUS === 2026/07/02 07:42:58 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/02 07:42:58 === DSC STATUS === 2026/07/02 07:42:58 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/02 07:42:58 === RECENT EVENTS === 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-njl6n 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-controller-5dd7d48f87-t2zf8: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-t2zf8 2026/07/02 07:42:58 07:42:17 WARN Pod/postgres-69fb4594cf-vlkn9: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:17 WARN Pod/konnectivity-agent-b6bsq: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:17 WARN Pod/global-pull-secret-syncer-569jc: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-webhook-5f8768dbfd-ddfp2: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-ddfp2 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-fhxd9 2026/07/02 07:42:58 07:42:16 WARN Pod/kube-apiserver-proxy-ip-10-0-130-228.ec2.internal: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:10 WARN Pod/spark-operator-controller-5dd7d48f87-t2zf8: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:10 WARN Pod/spark-operator-webhook-5f8768dbfd-fhxd9: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/global-pull-secret-syncer-ldz9z: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/spark-operator-webhook-5f8768dbfd-ddfp2: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/kube-apiserver-proxy-ip-10-0-134-81.ec2.internal: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/konnectivity-agent-mqjqx: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:03 WARN Pod/global-pull-secret-syncer-zrkbn: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:03 WARN Pod/spark-operator-controller-5dd7d48f87-njl6n: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:02 WARN Pod/konnectivity-agent-44png: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:02 WARN Pod/kube-apiserver-proxy-ip-10-0-136-131.ec2.internal: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:41:14 INFO Lease/cert-manager-controller: LeaderElection - cert-manager-59c77684cc-v45j9-external-cert-manager-controller stopped leading 2026/07/02 07:42:58 07:41:13 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkDriverCompleted - Driver scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver completed 2026/07/02 07:42:58 07:41:11 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkExecutorCompleted - Executor spark-pi-26d30b9f21c60382-exec-1 completed 2026/07/02 07:42:58 07:41:11 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/02 07:42:58 07:40:57 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkExecutorRunning - Executor spark-pi-26d30b9f21c60382-exec-1 is running 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-26d30b9f21c60382-exec-1 to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:40:56 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkExecutorPending - Executor spark-pi-26d30b9f21c60382-exec-1 is pending 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Started - Started container spark-kubernetes-executor 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: AddedInterface - Add eth0 [10.134.0.83/23] from ovn-kubernetes 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/02 07:42:58 07:40:47 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkDriverRunning - Driver scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver is running 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Started - Started container spark-kubernetes-driver 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Created - Created container: spark-kubernetes-driver 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: AddedInterface - Add eth0 [10.134.0.82/23] from ovn-kubernetes 2026/07/02 07:42:58 07:40:46 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Scheduled - Successfully assigned redhat-ods-applications/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:40:46 WARN Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-6395959f21c5e0d4-conf-map" not found 2026/07/02 07:42:58 07:40:46 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkApplicationSubmitted - SparkApplication scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717 was submitted successfully 2026/07/02 07:42:58 07:40:46 WARN Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-ad90a89f21c5ce81-driver-podspec-conf-map" not found 2026/07/02 07:42:58 07:39:56 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkDriverCompleted - Driver spark-pi-d931b31sidhc74ll6br0-driver completed 2026/07/02 07:42:58 07:39:53 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/02 07:42:58 07:39:53 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkExecutorCompleted - Executor spark-pi-c449739f21c4d30b-exec-1 completed 2026/07/02 07:42:58 07:39:39 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkExecutorRunning - Executor spark-pi-c449739f21c4d30b-exec-1 is running 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Started - Started container spark-kubernetes-executor 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: AddedInterface - Add eth0 [10.134.0.81/23] from ovn-kubernetes 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/02 07:42:58 07:39:37 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-c449739f21c4d30b-exec-1 to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:39:37 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkExecutorPending - Executor spark-pi-c449739f21c4d30b-exec-1 is pending 2026/07/02 07:42:58 07:39:30 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkDriverRunning - Driver spark-pi-d931b31sidhc74ll6br0-driver is running 2026/07/02 07:42:58 07:39:29 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Pulled - Successfully pulled image "quay.io/opendatahub/data-processing:Spark-v4.0.1" in 22.419s (22.419s including waiting). Image size: 1778928657 bytes. 2026/07/02 07:42:58 07:39:29 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Started - Started container spark-kubernetes-driver 2026/07/02 07:42:58 07:39:29 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Created - Created container: spark-kubernetes-driver 2026/07/02 07:42:58 07:39:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Created - Created container: controller 2026/07/02 07:42:58 07:39:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Pulled - Successfully pulled image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" in 1m4.343s (1m4.343s including waiting). Image size: 2736477781 bytes. 2026/07/02 07:42:58 07:39:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Started - Started container controller 2026/07/02 07:42:58 07:39:06 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-d931b31sidhc74ll6br0-driver to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:39:06 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Pulling - Pulling image "quay.io/opendatahub/data-processing:Spark-v4.0.1" 2026/07/02 07:42:58 07:39:06 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: AddedInterface - Add eth0 [10.134.0.80/23] from ovn-kubernetes 2026/07/02 07:42:58 07:39:06 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkApplicationSubmitted - SparkApplication spark-pi-d931b31sidhc74ll6br0 was submitted successfully 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-fhxd9 to ip-10-0-134-81.ec2.internal 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: AddedInterface - Add eth0 [10.133.0.151/23] from ovn-kubernetes 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Created - Created container: webhook 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Started - Started container webhook 2026/07/02 07:42:58 07:38:32 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-fhxd9 2026/07/02 07:42:58 07:38:32 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 1 to 2 2026/07/02 07:42:58 07:38:13 INFO Lease/spark-operator-webhook-lock: LeaderElection - spark-operator-webhook-5f8768dbfd-ddfp2_4b5d8c5e-db00-4781-ba61-6abd8706c068 became leader 2026/07/02 07:42:58 07:38:13 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Pulling - Pulling image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" 2026/07/02 07:42:58 07:38:13 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: AddedInterface - Add eth0 [10.134.0.79/23] from ovn-kubernetes 2026/07/02 07:42:58 07:38:12 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-njl6n to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:38:12 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 1 to 2 2026/07/02 07:42:58 07:38:12 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-njl6n 2026/07/02 07:42:58 07:38:11 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-5dd7d48f87-t2zf8_87362461-b81e-4570-8a17-e0a190c38fc9 became leader 2026/07/02 07:42:58 === RESOURCE QUOTAS === 2026/07/02 07:42:58 No resource quotas found in monitored namespaces 2026/07/02 07:42:58 Classification for group 1: infrastructure/storage (code=1006, confidence=medium) 2026/07/02 07:42:58 Evidence: event Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-6395959f21c5e0d4-conf-map" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume \"spark-conf-volume-driver\" : configmap \"spark-drv-6395959f21c5e0d4-conf-map\" not found"],"confidence":"medium"} 2026/07/02 07:42:58 Diagnostics complete for failed test: group 1 controller_test.go:763: Stopping: group 1 test failed. 2026/07/02 07:42:58 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/02 07:42:58 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/02 07:42:58 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (306.50s) === RUN TestOdhOperator 2026/07/02 07:37:21 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/02 07:42:58 TEST FAILURE DETECTED: components 2026/07/02 07:42:58 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/02 07:42:58 clusterhealth: node metrics unavailable (Metrics Server may not be installed): the server is currently unable to handle the request (get nodes.metrics.k8s.io) 2026/07/02 07:42:58 === CLUSTER STATE === 2026/07/02 07:42:58 Failed to collect node data: unhealthy nodes: ip-10-0-136-131.ec2.internal (Ready=Unknown) 2026/07/02 07:42:58 === DEPLOYMENTS === 2026/07/02 07:42:58 Failed to collect deployment data: deployments not ready: redhat-ods-applications/postgres (0/1), redhat-ods-applications/spark-operator-controller (0/2), redhat-ods-applications/spark-operator-webhook (0/2) 2026/07/02 07:42:58 === PODS === 2026/07/02 07:42:58 Namespace kube-system: no problematic pods found 2026/07/02 07:42:58 Namespace redhat-ods-applications: no problematic pods found 2026/07/02 07:42:58 === OPERATOR STATUS === 2026/07/02 07:42:58 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; dependent kueue: deployment 0/2 ready; dependent jobset: deployment 0/1 ready; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; dependent kuadrant: deployment 0/1 ready 2026/07/02 07:42:58 === DSCI STATUS === 2026/07/02 07:42:58 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/02 07:42:58 === DSC STATUS === 2026/07/02 07:42:58 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/02 07:42:58 === RECENT EVENTS === 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-njl6n 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-controller-5dd7d48f87-t2zf8: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-controller-5dd7d48f87-t2zf8 2026/07/02 07:42:58 07:42:17 WARN Pod/postgres-69fb4594cf-vlkn9: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:17 WARN Pod/konnectivity-agent-b6bsq: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:17 WARN Pod/global-pull-secret-syncer-569jc: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-webhook-5f8768dbfd-ddfp2: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-ddfp2 2026/07/02 07:42:58 07:42:17 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/spark-operator-webhook-5f8768dbfd-fhxd9 2026/07/02 07:42:58 07:42:16 WARN Pod/kube-apiserver-proxy-ip-10-0-130-228.ec2.internal: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:10 WARN Pod/spark-operator-controller-5dd7d48f87-t2zf8: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:10 WARN Pod/spark-operator-webhook-5f8768dbfd-fhxd9: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/global-pull-secret-syncer-ldz9z: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/spark-operator-webhook-5f8768dbfd-ddfp2: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/kube-apiserver-proxy-ip-10-0-134-81.ec2.internal: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:09 WARN Pod/konnectivity-agent-mqjqx: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:03 WARN Pod/global-pull-secret-syncer-zrkbn: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:03 WARN Pod/spark-operator-controller-5dd7d48f87-njl6n: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:02 WARN Pod/konnectivity-agent-44png: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:42:02 WARN Pod/kube-apiserver-proxy-ip-10-0-136-131.ec2.internal: NodeNotReady - Node is not ready 2026/07/02 07:42:58 07:41:14 INFO Lease/cert-manager-controller: LeaderElection - cert-manager-59c77684cc-v45j9-external-cert-manager-controller stopped leading 2026/07/02 07:42:58 07:41:13 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkDriverCompleted - Driver scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver completed 2026/07/02 07:42:58 07:41:11 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkExecutorCompleted - Executor spark-pi-26d30b9f21c60382-exec-1 completed 2026/07/02 07:42:58 07:41:11 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/02 07:42:58 07:40:57 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkExecutorRunning - Executor spark-pi-26d30b9f21c60382-exec-1 is running 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-26d30b9f21c60382-exec-1 to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:40:56 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkExecutorPending - Executor spark-pi-26d30b9f21c60382-exec-1 is pending 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Started - Started container spark-kubernetes-executor 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/02 07:42:58 07:40:56 INFO Pod/spark-pi-26d30b9f21c60382-exec-1: AddedInterface - Add eth0 [10.134.0.83/23] from ovn-kubernetes 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/02 07:42:58 07:40:47 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkDriverRunning - Driver scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver is running 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Started - Started container spark-kubernetes-driver 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Created - Created container: spark-kubernetes-driver 2026/07/02 07:42:58 07:40:47 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: AddedInterface - Add eth0 [10.134.0.82/23] from ovn-kubernetes 2026/07/02 07:42:58 07:40:46 INFO Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: Scheduled - Successfully assigned redhat-ods-applications/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:40:46 WARN Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-6395959f21c5e0d4-conf-map" not found 2026/07/02 07:42:58 07:40:46 INFO SparkApplication/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717: SparkApplicationSubmitted - SparkApplication scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717 was submitted successfully 2026/07/02 07:42:58 07:40:46 WARN Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-ad90a89f21c5ce81-driver-podspec-conf-map" not found 2026/07/02 07:42:58 07:39:56 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkDriverCompleted - Driver spark-pi-d931b31sidhc74ll6br0-driver completed 2026/07/02 07:42:58 07:39:53 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/02 07:42:58 07:39:53 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkExecutorCompleted - Executor spark-pi-c449739f21c4d30b-exec-1 completed 2026/07/02 07:42:58 07:39:39 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkExecutorRunning - Executor spark-pi-c449739f21c4d30b-exec-1 is running 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Started - Started container spark-kubernetes-executor 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: AddedInterface - Add eth0 [10.134.0.81/23] from ovn-kubernetes 2026/07/02 07:42:58 07:39:38 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/02 07:42:58 07:39:37 INFO Pod/spark-pi-c449739f21c4d30b-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-c449739f21c4d30b-exec-1 to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:39:37 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkExecutorPending - Executor spark-pi-c449739f21c4d30b-exec-1 is pending 2026/07/02 07:42:58 07:39:30 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkDriverRunning - Driver spark-pi-d931b31sidhc74ll6br0-driver is running 2026/07/02 07:42:58 07:39:29 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Pulled - Successfully pulled image "quay.io/opendatahub/data-processing:Spark-v4.0.1" in 22.419s (22.419s including waiting). Image size: 1778928657 bytes. 2026/07/02 07:42:58 07:39:29 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Started - Started container spark-kubernetes-driver 2026/07/02 07:42:58 07:39:29 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Created - Created container: spark-kubernetes-driver 2026/07/02 07:42:58 07:39:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Created - Created container: controller 2026/07/02 07:42:58 07:39:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Pulled - Successfully pulled image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" in 1m4.343s (1m4.343s including waiting). Image size: 2736477781 bytes. 2026/07/02 07:42:58 07:39:17 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Started - Started container controller 2026/07/02 07:42:58 07:39:06 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-d931b31sidhc74ll6br0-driver to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:39:06 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: Pulling - Pulling image "quay.io/opendatahub/data-processing:Spark-v4.0.1" 2026/07/02 07:42:58 07:39:06 INFO Pod/spark-pi-d931b31sidhc74ll6br0-driver: AddedInterface - Add eth0 [10.134.0.80/23] from ovn-kubernetes 2026/07/02 07:42:58 07:39:06 INFO SparkApplication/spark-pi-d931b31sidhc74ll6br0: SparkApplicationSubmitted - SparkApplication spark-pi-d931b31sidhc74ll6br0 was submitted successfully 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-fhxd9 to ip-10-0-134-81.ec2.internal 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: AddedInterface - Add eth0 [10.133.0.151/23] from ovn-kubernetes 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Created - Created container: webhook 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/02 07:42:58 07:38:32 INFO Pod/spark-operator-webhook-5f8768dbfd-fhxd9: Started - Started container webhook 2026/07/02 07:42:58 07:38:32 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-fhxd9 2026/07/02 07:42:58 07:38:32 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 1 to 2 2026/07/02 07:42:58 07:38:13 INFO Lease/spark-operator-webhook-lock: LeaderElection - spark-operator-webhook-5f8768dbfd-ddfp2_4b5d8c5e-db00-4781-ba61-6abd8706c068 became leader 2026/07/02 07:42:58 07:38:13 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Pulling - Pulling image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" 2026/07/02 07:42:58 07:38:13 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: AddedInterface - Add eth0 [10.134.0.79/23] from ovn-kubernetes 2026/07/02 07:42:58 07:38:12 INFO Pod/spark-operator-controller-5dd7d48f87-njl6n: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-njl6n to ip-10-0-136-131.ec2.internal 2026/07/02 07:42:58 07:38:12 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 1 to 2 2026/07/02 07:42:58 07:38:12 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-njl6n 2026/07/02 07:42:58 07:38:11 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-5dd7d48f87-t2zf8_87362461-b81e-4570-8a17-e0a190c38fc9 became leader 2026/07/02 07:42:58 === RESOURCE QUOTAS === 2026/07/02 07:42:58 No resource quotas found in monitored namespaces 2026/07/02 07:42:58 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/07/02 07:42:58 Evidence: event Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-6395959f21c5e0d4-conf-map" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/scheduled-spark-pi-d931bkhsidhc74ll6brg-1782978032001268717-driver: FailedMount - MountVolume.SetUp failed for volume \"spark-conf-volume-driver\" : configmap \"spark-drv-6395959f21c5e0d4-conf-map\" not found"],"confidence":"medium"} 2026/07/02 07:42:58 Diagnostics complete for failed test: components controller_test.go:491: Stopping: components test failed. 2026/07/02 07:42:58 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/02 07:42:58 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/02 07:42:59 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (338.16s)