> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 10/30/26 19:56:26.144 < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 10/30/26 19:56:26.146 (2ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 19:56:26.146 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 19:56:26.278 (132ms) > Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:84 @ 10/30/26 19:56:26.278 STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:93 @ 10/30/26 19:56:26.278 END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:93 @ 10/30/26 19:56:26.359 (81ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:98 @ 10/30/26 19:56:26.359 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:98 @ 10/30/26 19:56:26.642 (283ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:116 @ 10/30/26 19:56:26.642 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:116 @ 10/30/26 20:04:49.771 (8m23.129s) < Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:84 @ 10/30/26 20:04:49.771 (8m23.493s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:04:49.771 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:04:49.791 (20ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:04:49.791 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:04:49.82 (29ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:145 @ 10/30/26 20:04:49.82 [SKIPPED] MPI runtime image is not available In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:147 @ 10/30/26 20:04:49.821 < Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:145 @ 10/30/26 20:04:49.821 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:04:49.821 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:04:49.843 (22ms) [FAILED] Expected success, but got an error: <*errors.StatusError | 0x380ff807ea00>: admission webhook "validator.trainjob.trainer.kubeflow.org" denied the request: spec.runtimeRef: Invalid value: {"name":"deepspeed-distributed","apiGroup":"trainer.kubeflow.org","kind":"ClusterTrainingRuntime"}: ClusterTrainingRuntime.trainer.kubeflow.org "deepspeed-distributed" not found: specified clusterTrainingRuntime must be created before the TrainJob is created { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "admission webhook \"validator.trainjob.trainer.kubeflow.org\" denied the request: spec.runtimeRef: Invalid value: {\"name\":\"deepspeed-distributed\",\"apiGroup\":\"trainer.kubeflow.org\",\"kind\":\"ClusterTrainingRuntime\"}: ClusterTrainingRuntime.trainer.kubeflow.org \"deepspeed-distributed\" not found: specified clusterTrainingRuntime must be created before the TrainJob is created", Reason: "Forbidden", Details: nil, Code: 403, }, } In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:216 @ 10/30/26 20:04:49.943 > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:04:49.843 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:04:49.879 (36ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference and OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:203 @ 10/30/26 20:04:49.879 STEP: Create a TrainJob with deepspeed-distributed runtime reference and the OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:215 @ 10/30/26 20:04:49.879 END STEP: Create a TrainJob with deepspeed-distributed runtime reference and the OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:215 @ 10/30/26 20:04:49.943 (64ms) [FAILED] Expected success, but got an error: <*errors.StatusError | 0x380ff807ea00>: admission webhook "validator.trainjob.trainer.kubeflow.org" denied the request: spec.runtimeRef: Invalid value: {"name":"deepspeed-distributed","apiGroup":"trainer.kubeflow.org","kind":"ClusterTrainingRuntime"}: ClusterTrainingRuntime.trainer.kubeflow.org "deepspeed-distributed" not found: specified clusterTrainingRuntime must be created before the TrainJob is created { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "admission webhook \"validator.trainjob.trainer.kubeflow.org\" denied the request: spec.runtimeRef: Invalid value: {\"name\":\"deepspeed-distributed\",\"apiGroup\":\"trainer.kubeflow.org\",\"kind\":\"ClusterTrainingRuntime\"}: ClusterTrainingRuntime.trainer.kubeflow.org \"deepspeed-distributed\" not found: specified clusterTrainingRuntime must be created before the TrainJob is created", Reason: "Forbidden", Details: nil, Code: 403, }, } In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:216 @ 10/30/26 20:04:49.943 < Exit [It] should create TrainJob with DeepSpeed runtime reference and OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:203 @ 10/30/26 20:04:49.943 (64ms) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:04:49.943 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:04:49.961 (18ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:04:49.961 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:04:49.993 (32ms) > Enter [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:259 @ 10/30/26 20:04:49.993 STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:265 @ 10/30/26 20:04:49.993 END STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:265 @ 10/30/26 20:04:50.019 (25ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:270 @ 10/30/26 20:04:50.019 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:270 @ 10/30/26 20:04:50.295 (276ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:288 @ 10/30/26 20:04:50.295 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:288 @ 10/30/26 20:07:18.599 (2m28.304s) < Exit [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:259 @ 10/30/26 20:07:18.599 (2m28.605s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:07:18.599 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:07:18.622 (23ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:07:18.622 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:07:18.652 (30ms) > Enter [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:317 @ 10/30/26 20:07:18.652 STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:323 @ 10/30/26 20:07:18.652 END STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:323 @ 10/30/26 20:07:18.718 (66ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:328 @ 10/30/26 20:07:18.718 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:328 @ 10/30/26 20:07:18.994 (276ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:346 @ 10/30/26 20:07:18.994 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:346 @ 10/30/26 20:08:07.89 (48.896s) < Exit [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:317 @ 10/30/26 20:08:07.89 (49.238s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:08:07.89 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:08:07.906 (15ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:08:07.906 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:08:07.934 (28ms) > Enter [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:374 @ 10/30/26 20:08:07.934 STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:393 @ 10/30/26 20:08:07.934 END STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:393 @ 10/30/26 20:08:07.957 (23ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:397 @ 10/30/26 20:08:07.957 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:397 @ 10/30/26 20:08:08.241 (284ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:414 @ 10/30/26 20:08:08.241 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:414 @ 10/30/26 20:08:38.366 (30.125s) < Exit [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:374 @ 10/30/26 20:08:38.366 (30.432s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:08:38.366 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:08:38.382 (16ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:08:38.382 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:08:38.412 (31ms) > Enter [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:442 @ 10/30/26 20:08:38.412 STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:494 @ 10/30/26 20:08:38.412 END STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:494 @ 10/30/26 20:08:38.437 (24ms) STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:498 @ 10/30/26 20:08:38.437 END STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:498 @ 10/30/26 20:08:38.452 (15ms) < Exit [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:442 @ 10/30/26 20:08:38.452 (39ms) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:08:38.452 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:08:38.469 (18ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:08:38.469 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:08:38.498 (28ms) > Enter [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:512 @ 10/30/26 20:08:38.498 STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:566 @ 10/30/26 20:08:38.498 END STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:566 @ 10/30/26 20:08:38.527 (29ms) STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:571 @ 10/30/26 20:08:38.527 END STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:571 @ 10/30/26 20:08:38.541 (14ms) STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:588 @ 10/30/26 20:08:39.541 END STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:588 @ 10/30/26 20:08:39.602 (60ms) STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:607 @ 10/30/26 20:08:39.602 END STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:607 @ 10/30/26 20:08:39.617 (15ms) < Exit [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:512 @ 10/30/26 20:08:39.617 (1.119s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:08:39.617 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:08:39.64 (23ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:08:39.64 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:08:39.669 (29ms) > Enter [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:629 @ 10/30/26 20:08:39.669 STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:636 @ 10/30/26 20:08:39.669 END STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:636 @ 10/30/26 20:08:39.698 (30ms) STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:640 @ 10/30/26 20:08:39.698 END STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:640 @ 10/30/26 20:08:54.699 (15.001s) < Exit [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:629 @ 10/30/26 20:08:54.699 (15.031s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:08:54.699 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:08:54.74 (40ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:08:54.74 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:08:54.791 (50ms) > Enter [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:652 @ 10/30/26 20:08:54.791 STEP: Waiting for TrainJob to fail due to deadline - /workspace/source/src_code/test/e2e/e2e_test.go:667 @ 10/30/26 20:08:54.825 STEP: Ensuring the underlying JobSet is deleted - /workspace/source/src_code/test/e2e/e2e_test.go:674 @ 10/30/26 20:09:04.084 < Exit [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:652 @ 10/30/26 20:09:04.097 (9.306s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:09:04.097 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:09:04.115 (18ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:09:04.115 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:09:04.15 (34ms) > Enter [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:905 @ 10/30/26 20:09:04.15 STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 10/30/26 20:09:04.15 END STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 10/30/26 20:09:04.197 (47ms) STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 10/30/26 20:09:04.197 END STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 10/30/26 20:09:04.219 (22ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 10/30/26 20:09:04.219 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 10/30/26 20:09:04.516 (297ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 10/30/26 20:09:04.516 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 10/30/26 20:09:04.552 (36ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 10/30/26 20:09:04.552 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 10/30/26 20:09:04.828 (276ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 10/30/26 20:09:04.828 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 10/30/26 20:09:04.862 (34ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 10/30/26 20:09:04.862 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 10/30/26 20:09:04.908 (46ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 10/30/26 20:09:04.908 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 10/30/26 20:09:05.185 (277ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 10/30/26 20:09:05.185 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 10/30/26 20:09:05.228 (44ms) < Exit [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:905 @ 10/30/26 20:09:05.228 (1.079s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:09:05.228 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:09:05.246 (18ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 10/30/26 20:09:05.247 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 10/30/26 20:09:05.264 (18ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:09:05.265 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:09:05.292 (28ms) > Enter [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:917 @ 10/30/26 20:09:05.292 STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 10/30/26 20:09:05.292 END STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 10/30/26 20:09:05.336 (44ms) STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 10/30/26 20:09:05.337 END STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 10/30/26 20:09:05.358 (21ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 10/30/26 20:09:05.358 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 10/30/26 20:09:05.635 (277ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 10/30/26 20:09:05.635 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 10/30/26 20:09:05.674 (39ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 10/30/26 20:09:05.674 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 10/30/26 20:09:05.95 (277ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 10/30/26 20:09:05.951 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 10/30/26 20:09:05.984 (33ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 10/30/26 20:09:05.984 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 10/30/26 20:09:06.022 (38ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 10/30/26 20:09:06.022 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 10/30/26 20:09:06.299 (277ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 10/30/26 20:09:06.299 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 10/30/26 20:09:06.343 (44ms) < Exit [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:917 @ 10/30/26 20:09:06.343 (1.05s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:09:06.343 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:09:06.358 (15ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 10/30/26 20:09:06.358 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 10/30/26 20:09:06.375 (18ms) === RUN TestKubeflowSdk fashion_mnist_tests.go:56: Created User RoleBinding test-ns-tdnz6/rb-xnlvv successfully fashion_mnist_tests.go:57: Created User RoleBinding test-ns-tdnz6/rb-7rdkg successfully fashion_mnist_tests.go:57: Created ClusterRole /clusterrole-chpc2 successfully fashion_mnist_tests.go:57: Created User ClusterRoleBinding crb-b5trf successfully fashion_mnist_tests.go:64: Created ConfigMap test-ns-tdnz6/config-8ngq6 successfully fashion_mnist_tests.go:82: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX. fashion_mnist_tests.go:84: Created PersistentVolumeClaim test-ns-tdnz6/pvc-n7pzh successfully fashion_mnist_tests.go:93: OpenShift API URL: https://a97f8398d4dc944878cfd92eb7b75d4e-7a4ecc4a4c388e7e.elb.us-east-2.amazonaws.com:6443 dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1140 notebook.go:44: Created test Deployment test-ns-tdnz6/dw-deployment-f8sq7 test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdk1481802450 fashion_mnist_tests.go:138: Deployment runner execution reported FAILURE Unexpected error: <*errors.errorString | 0x2fdce6f4f3d0>: pod execution failed { s: "pod execution failed", } occurred test.go:171: Retrieving Pod Container test-ns-tdnz6/dw-deployment-f8sq7-797844fc4b-7mt2w/notebook logs --- FAIL: TestKubeflowSdk (13.70s) === RUN TestKubeflowSdkKueueIntegration kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK kueue_operator.go:227: Kueue CR 'cluster' exists kueue_operator.go:229: Waiting for Kueue CR to be ready... kueue_operator.go:236: Kueue CR is ready kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR... kueue_operator.go:243: TrainJob framework is present in Kueue CR kueue_operator.go:239: Verifying Deployment framework is present in Kueue CR... kueue_operator.go:243: Deployment framework is present in Kueue CR fashion_mnist_tests.go:148: Created Kueue-managed namespace: test-ns-jxhrf fashion_mnist_tests.go:153: Created User RoleBinding test-ns-jxhrf/rb-7vvlx successfully fashion_mnist_tests.go:154: Created User RoleBinding test-ns-jxhrf/rb-2vjbp successfully fashion_mnist_tests.go:154: Created ClusterRole /clusterrole-h4lq6 successfully fashion_mnist_tests.go:154: Created User ClusterRoleBinding crb-jw2pt successfully fashion_mnist_tests.go:157: Created Kueue ResourceFlavor rf-5kg2d successfully fashion_mnist_tests.go:184: Created Kueue ClusterQueue cq-ssz5w successfully fashion_mnist_tests.go:189: Created Kueue LocalQueue test-ns-jxhrf/lq-zkkb9 successfully fashion_mnist_tests.go:190: Created custom LocalQueue lq-zkkb9 for TrainJob fashion_mnist_tests.go:197: Created ConfigMap test-ns-jxhrf/config-8mpd4 successfully fashion_mnist_tests.go:215: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX. fashion_mnist_tests.go:217: Created PersistentVolumeClaim test-ns-jxhrf/pvc-xgpvx successfully fashion_mnist_tests.go:226: OpenShift API URL: https://a97f8398d4dc944878cfd92eb7b75d4e-7a4ecc4a4c388e7e.elb.us-east-2.amazonaws.com:6443 dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1140 notebook.go:44: Created test Deployment test-ns-jxhrf/dw-deployment-hcnpm fashion_mnist_tests.go:271: Verifying SDK-submitted TrainJob has custom queue label: lq-zkkb9 test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration3181399757 fashion_mnist_tests.go:272: Timed out after 1200.001s. Expected <[]v1alpha1.TrainJob | len:0, cap:0>: [] to have length 1 test.go:171: Retrieving Pod Container test-ns-jxhrf/dw-deployment-hcnpm-58d569f9dd-r225h/notebook logs kueue_operator.go:100: Collecting Kueue diagnostics... test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration1541199642 kueue_operator.go:100: Stored Kueue CR state kueue_operator.go:100: DataScienceCluster is not installed; skipping DSC diagnostics kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-6f9bf978cc-l6rsc/manager kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-6f9bf978cc-z8b62/manager --- FAIL: TestKubeflowSdkKueueIntegration (1200.97s)