> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 10/30/26 20:12:35.407 < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 10/30/26 20:12:35.409 (1ms) [FAILED] Timed out after 600.000s. The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:120 with: Expected object to be comparable, diff: []v1.Condition( - nil, + { + { + Type: "Complete", + Status: "True", + Reason: "AllJobsCompleted", + Message: "jobset completed successfully", + }, + }, ) In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:138 @ 10/30/26 20:22:35.941 > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:12:35.409 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:12:35.576 (167ms) > Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:84 @ 10/30/26 20:12:35.576 STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:93 @ 10/30/26 20:12:35.577 END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:93 @ 10/30/26 20:12:35.652 (76ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:98 @ 10/30/26 20:12:35.652 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:98 @ 10/30/26 20:12:35.94 (288ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:116 @ 10/30/26 20:12:35.94 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:116 @ 10/30/26 20:22:35.941 (10m0.001s) [FAILED] Timed out after 600.000s. The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:120 with: Expected object to be comparable, diff: []v1.Condition( - nil, + { + { + Type: "Complete", + Status: "True", + Reason: "AllJobsCompleted", + Message: "jobset completed successfully", + }, + }, ) In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:138 @ 10/30/26 20:22:35.941 < Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:84 @ 10/30/26 20:22:35.941 (10m0.365s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:22:35.941 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:22:35.962 (21ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:22:35.962 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:22:36.002 (39ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:145 @ 10/30/26 20:22:36.002 [SKIPPED] MPI runtime image is not available In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:147 @ 10/30/26 20:22:36.002 < Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:145 @ 10/30/26 20:22:36.002 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:22:36.002 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:22:36.022 (20ms) [FAILED] Expected success, but got an error: <*errors.StatusError | 0x24215ef16e60>: admission webhook "validator.trainjob.trainer.kubeflow.org" denied the request: spec.runtimeRef: Invalid value: {"name":"deepspeed-distributed","apiGroup":"trainer.kubeflow.org","kind":"ClusterTrainingRuntime"}: ClusterTrainingRuntime.trainer.kubeflow.org "deepspeed-distributed" not found: specified clusterTrainingRuntime must be created before the TrainJob is created { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "admission webhook \"validator.trainjob.trainer.kubeflow.org\" denied the request: spec.runtimeRef: Invalid value: {\"name\":\"deepspeed-distributed\",\"apiGroup\":\"trainer.kubeflow.org\",\"kind\":\"ClusterTrainingRuntime\"}: ClusterTrainingRuntime.trainer.kubeflow.org \"deepspeed-distributed\" not found: specified clusterTrainingRuntime must be created before the TrainJob is created", Reason: "Forbidden", Details: nil, Code: 403, }, } In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:216 @ 10/30/26 20:22:36.13 > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:22:36.022 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:22:36.061 (38ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference and OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:203 @ 10/30/26 20:22:36.061 STEP: Create a TrainJob with deepspeed-distributed runtime reference and the OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:215 @ 10/30/26 20:22:36.061 END STEP: Create a TrainJob with deepspeed-distributed runtime reference and the OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:215 @ 10/30/26 20:22:36.129 (69ms) [FAILED] Expected success, but got an error: <*errors.StatusError | 0x24215ef16e60>: admission webhook "validator.trainjob.trainer.kubeflow.org" denied the request: spec.runtimeRef: Invalid value: {"name":"deepspeed-distributed","apiGroup":"trainer.kubeflow.org","kind":"ClusterTrainingRuntime"}: ClusterTrainingRuntime.trainer.kubeflow.org "deepspeed-distributed" not found: specified clusterTrainingRuntime must be created before the TrainJob is created { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "admission webhook \"validator.trainjob.trainer.kubeflow.org\" denied the request: spec.runtimeRef: Invalid value: {\"name\":\"deepspeed-distributed\",\"apiGroup\":\"trainer.kubeflow.org\",\"kind\":\"ClusterTrainingRuntime\"}: ClusterTrainingRuntime.trainer.kubeflow.org \"deepspeed-distributed\" not found: specified clusterTrainingRuntime must be created before the TrainJob is created", Reason: "Forbidden", Details: nil, Code: 403, }, } In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:216 @ 10/30/26 20:22:36.13 < Exit [It] should create TrainJob with DeepSpeed runtime reference and OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:203 @ 10/30/26 20:22:36.13 (69ms) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:22:36.13 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:22:36.15 (20ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:22:36.15 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:22:36.19 (40ms) > Enter [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:259 @ 10/30/26 20:22:36.19 STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:265 @ 10/30/26 20:22:36.19 END STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:265 @ 10/30/26 20:22:36.216 (27ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:270 @ 10/30/26 20:22:36.216 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:270 @ 10/30/26 20:22:36.503 (286ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:288 @ 10/30/26 20:22:36.503 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:288 @ 10/30/26 20:24:32.13 (1m55.627s) < Exit [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:259 @ 10/30/26 20:24:32.13 (1m55.941s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:24:32.13 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:24:32.151 (20ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:24:32.151 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:24:32.191 (40ms) > Enter [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:317 @ 10/30/26 20:24:32.191 STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:323 @ 10/30/26 20:24:32.191 END STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:323 @ 10/30/26 20:24:32.265 (74ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:328 @ 10/30/26 20:24:32.265 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:328 @ 10/30/26 20:24:32.561 (296ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:346 @ 10/30/26 20:24:32.561 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:346 @ 10/30/26 20:26:32.366 (1m59.805s) < Exit [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:317 @ 10/30/26 20:26:32.366 (2m0.175s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:26:32.366 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:26:32.401 (35ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:26:32.402 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:26:32.452 (50ms) > Enter [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:374 @ 10/30/26 20:26:32.452 STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:393 @ 10/30/26 20:26:32.452 END STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:393 @ 10/30/26 20:26:32.532 (81ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:397 @ 10/30/26 20:26:32.533 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:397 @ 10/30/26 20:26:33.086 (554ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:414 @ 10/30/26 20:26:33.086 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:414 @ 10/30/26 20:26:59.382 (26.296s) < Exit [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:374 @ 10/30/26 20:26:59.382 (26.931s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:26:59.382 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:26:59.402 (19ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:26:59.402 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:26:59.439 (37ms) > Enter [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:442 @ 10/30/26 20:26:59.439 STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:494 @ 10/30/26 20:26:59.439 END STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:494 @ 10/30/26 20:26:59.477 (38ms) STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:498 @ 10/30/26 20:26:59.477 END STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:498 @ 10/30/26 20:26:59.494 (18ms) < Exit [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:442 @ 10/30/26 20:26:59.494 (56ms) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:26:59.494 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:26:59.513 (19ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:26:59.513 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:26:59.558 (45ms) > Enter [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:512 @ 10/30/26 20:26:59.558 STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:566 @ 10/30/26 20:26:59.558 END STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:566 @ 10/30/26 20:26:59.589 (31ms) STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:571 @ 10/30/26 20:26:59.589 END STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:571 @ 10/30/26 20:26:59.606 (17ms) STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:588 @ 10/30/26 20:27:00.606 END STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:588 @ 10/30/26 20:27:00.673 (67ms) STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:607 @ 10/30/26 20:27:00.673 END STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:607 @ 10/30/26 20:27:00.69 (17ms) < Exit [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:512 @ 10/30/26 20:27:00.69 (1.132s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:27:00.69 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:27:00.711 (21ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:27:00.712 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:27:00.749 (38ms) > Enter [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:629 @ 10/30/26 20:27:00.749 STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:636 @ 10/30/26 20:27:00.749 END STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:636 @ 10/30/26 20:27:00.779 (30ms) STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:640 @ 10/30/26 20:27:00.779 END STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:640 @ 10/30/26 20:27:15.779 (15s) < Exit [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:629 @ 10/30/26 20:27:15.779 (15.03s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:27:15.779 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:27:15.801 (21ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:27:15.801 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:27:15.84 (39ms) > Enter [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:652 @ 10/30/26 20:27:15.84 STEP: Waiting for TrainJob to fail due to deadline - /workspace/source/src_code/test/e2e/e2e_test.go:667 @ 10/30/26 20:27:15.868 STEP: Ensuring the underlying JobSet is deleted - /workspace/source/src_code/test/e2e/e2e_test.go:674 @ 10/30/26 20:27:25.029 < Exit [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:652 @ 10/30/26 20:27:25.046 (9.206s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:27:25.046 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:27:25.072 (27ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:27:25.072 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:27:25.109 (37ms) > Enter [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:905 @ 10/30/26 20:27:25.109 STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 10/30/26 20:27:25.11 END STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 10/30/26 20:27:25.168 (58ms) STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 10/30/26 20:27:25.168 END STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 10/30/26 20:27:25.195 (27ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 10/30/26 20:27:25.195 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 10/30/26 20:27:25.482 (286ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 10/30/26 20:27:25.482 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 10/30/26 20:27:25.532 (50ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 10/30/26 20:27:25.532 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 10/30/26 20:27:25.818 (286ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 10/30/26 20:27:25.818 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 10/30/26 20:27:25.862 (45ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 10/30/26 20:27:25.863 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 10/30/26 20:27:25.91 (47ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 10/30/26 20:27:25.91 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 10/30/26 20:27:26.195 (285ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 10/30/26 20:27:26.195 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 10/30/26 20:27:26.249 (54ms) < Exit [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:905 @ 10/30/26 20:27:26.249 (1.139s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:27:26.249 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:27:26.268 (19ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 10/30/26 20:27:26.268 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 10/30/26 20:27:26.288 (20ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:27:26.288 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 10/30/26 20:27:26.325 (36ms) > Enter [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:917 @ 10/30/26 20:27:26.325 STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 10/30/26 20:27:26.325 END STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 10/30/26 20:27:26.374 (49ms) STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 10/30/26 20:27:26.374 END STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 10/30/26 20:27:26.403 (29ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 10/30/26 20:27:26.403 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 10/30/26 20:27:26.687 (284ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 10/30/26 20:27:26.687 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 10/30/26 20:27:26.735 (47ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 10/30/26 20:27:26.735 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 10/30/26 20:27:27.02 (285ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 10/30/26 20:27:27.02 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 10/30/26 20:27:27.063 (43ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 10/30/26 20:27:27.063 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 10/30/26 20:27:27.11 (47ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 10/30/26 20:27:27.11 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 10/30/26 20:27:27.396 (286ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 10/30/26 20:27:27.396 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 10/30/26 20:27:27.738 (342ms) < Exit [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:917 @ 10/30/26 20:27:27.738 (1.413s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:27:27.738 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 10/30/26 20:27:27.758 (20ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 10/30/26 20:27:27.758 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 10/30/26 20:27:27.783 (25ms)