> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 11/03/26 06:07:12.462 < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 11/03/26 06:07:12.464 (2ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:07:12.464 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:07:12.588 (123ms) > Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:84 @ 11/03/26 06:07:12.588 STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:93 @ 11/03/26 06:07:12.588 END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:93 @ 11/03/26 06:07:12.769 (181ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:98 @ 11/03/26 06:07:12.769 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:98 @ 11/03/26 06:07:13.334 (565ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:116 @ 11/03/26 06:07:13.334 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:116 @ 11/03/26 06:11:39.132 (4m25.798s) < Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:84 @ 11/03/26 06:11:39.132 (4m26.544s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:11:39.132 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:11:39.148 (16ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:11:39.148 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:11:39.19 (41ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:145 @ 11/03/26 06:11:39.19 [SKIPPED] MPI runtime image is not available In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:147 @ 11/03/26 06:11:39.19 < Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:145 @ 11/03/26 06:11:39.19 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:11:39.19 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:11:39.216 (27ms) [FAILED] Expected success, but got an error: <*errors.StatusError | 0x2f011bc0b180>: admission webhook "validator.trainjob.trainer.kubeflow.org" denied the request: spec.runtimeRef: Invalid value: {"name":"deepspeed-distributed","apiGroup":"trainer.kubeflow.org","kind":"ClusterTrainingRuntime"}: ClusterTrainingRuntime.trainer.kubeflow.org "deepspeed-distributed" not found: specified clusterTrainingRuntime must be created before the TrainJob is created { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "admission webhook \"validator.trainjob.trainer.kubeflow.org\" denied the request: spec.runtimeRef: Invalid value: {\"name\":\"deepspeed-distributed\",\"apiGroup\":\"trainer.kubeflow.org\",\"kind\":\"ClusterTrainingRuntime\"}: ClusterTrainingRuntime.trainer.kubeflow.org \"deepspeed-distributed\" not found: specified clusterTrainingRuntime must be created before the TrainJob is created", Reason: "Forbidden", Details: nil, Code: 403, }, } In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:216 @ 11/03/26 06:11:39.405 > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:11:39.216 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:11:39.256 (39ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference and OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:203 @ 11/03/26 06:11:39.256 STEP: Create a TrainJob with deepspeed-distributed runtime reference and the OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:215 @ 11/03/26 06:11:39.256 END STEP: Create a TrainJob with deepspeed-distributed runtime reference and the OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:215 @ 11/03/26 06:11:39.404 (149ms) [FAILED] Expected success, but got an error: <*errors.StatusError | 0x2f011bc0b180>: admission webhook "validator.trainjob.trainer.kubeflow.org" denied the request: spec.runtimeRef: Invalid value: {"name":"deepspeed-distributed","apiGroup":"trainer.kubeflow.org","kind":"ClusterTrainingRuntime"}: ClusterTrainingRuntime.trainer.kubeflow.org "deepspeed-distributed" not found: specified clusterTrainingRuntime must be created before the TrainJob is created { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "admission webhook \"validator.trainjob.trainer.kubeflow.org\" denied the request: spec.runtimeRef: Invalid value: {\"name\":\"deepspeed-distributed\",\"apiGroup\":\"trainer.kubeflow.org\",\"kind\":\"ClusterTrainingRuntime\"}: ClusterTrainingRuntime.trainer.kubeflow.org \"deepspeed-distributed\" not found: specified clusterTrainingRuntime must be created before the TrainJob is created", Reason: "Forbidden", Details: nil, Code: 403, }, } In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:216 @ 11/03/26 06:11:39.405 < Exit [It] should create TrainJob with DeepSpeed runtime reference and OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:203 @ 11/03/26 06:11:39.405 (149ms) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:11:39.405 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:11:39.419 (14ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:11:39.419 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:11:39.454 (35ms) > Enter [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:259 @ 11/03/26 06:11:39.454 STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:265 @ 11/03/26 06:11:39.454 END STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:265 @ 11/03/26 06:11:39.515 (60ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:270 @ 11/03/26 06:11:39.515 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:270 @ 11/03/26 06:11:40.318 (803ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:288 @ 11/03/26 06:11:40.318 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:288 @ 11/03/26 06:14:12.687 (2m32.369s) < Exit [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:259 @ 11/03/26 06:14:12.687 (2m33.233s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:14:12.687 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:14:12.702 (14ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:14:12.702 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:14:12.731 (30ms) > Enter [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:317 @ 11/03/26 06:14:12.731 STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:323 @ 11/03/26 06:14:12.731 END STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:323 @ 11/03/26 06:14:12.864 (133ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:328 @ 11/03/26 06:14:12.864 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:328 @ 11/03/26 06:14:13.405 (541ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:346 @ 11/03/26 06:14:13.405 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:346 @ 11/03/26 06:14:57.85 (44.445s) < Exit [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:317 @ 11/03/26 06:14:57.85 (45.119s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:14:57.85 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:14:57.864 (14ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:14:57.865 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:14:57.892 (27ms) > Enter [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:374 @ 11/03/26 06:14:57.892 STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:393 @ 11/03/26 06:14:57.892 END STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:393 @ 11/03/26 06:14:57.94 (48ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:397 @ 11/03/26 06:14:57.94 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:397 @ 11/03/26 06:14:58.481 (540ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:414 @ 11/03/26 06:14:58.481 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:414 @ 11/03/26 06:15:29.777 (31.296s) < Exit [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:374 @ 11/03/26 06:15:29.777 (31.885s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:29.777 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:29.792 (15ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:15:29.792 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:15:29.824 (32ms) > Enter [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:442 @ 11/03/26 06:15:29.824 STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:494 @ 11/03/26 06:15:29.824 END STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:494 @ 11/03/26 06:15:29.874 (50ms) STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:498 @ 11/03/26 06:15:29.874 END STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:498 @ 11/03/26 06:15:29.886 (12ms) < Exit [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:442 @ 11/03/26 06:15:29.886 (62ms) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:29.886 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:29.902 (16ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:15:29.902 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:15:29.928 (26ms) > Enter [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:512 @ 11/03/26 06:15:29.928 STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:566 @ 11/03/26 06:15:29.928 END STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:566 @ 11/03/26 06:15:29.978 (50ms) STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:571 @ 11/03/26 06:15:29.978 END STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:571 @ 11/03/26 06:15:29.99 (12ms) STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:588 @ 11/03/26 06:15:30.99 END STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:588 @ 11/03/26 06:15:31.059 (69ms) STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:607 @ 11/03/26 06:15:31.059 END STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:607 @ 11/03/26 06:15:31.072 (13ms) < Exit [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:512 @ 11/03/26 06:15:31.072 (1.143s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:31.072 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:31.086 (14ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:15:31.086 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:15:31.115 (29ms) > Enter [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:629 @ 11/03/26 06:15:31.115 STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:636 @ 11/03/26 06:15:31.115 END STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:636 @ 11/03/26 06:15:31.169 (54ms) STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:640 @ 11/03/26 06:15:31.169 END STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:640 @ 11/03/26 06:15:46.169 (15s) < Exit [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:629 @ 11/03/26 06:15:46.169 (15.054s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:46.169 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:46.183 (14ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:15:46.183 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:15:46.214 (31ms) > Enter [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:652 @ 11/03/26 06:15:46.214 STEP: Waiting for TrainJob to fail due to deadline - /workspace/source/src_code/test/e2e/e2e_test.go:667 @ 11/03/26 06:15:46.262 STEP: Ensuring the underlying JobSet is deleted - /workspace/source/src_code/test/e2e/e2e_test.go:674 @ 11/03/26 06:15:56.26 < Exit [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:652 @ 11/03/26 06:15:56.272 (10.058s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:56.272 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:56.286 (14ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:15:56.286 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:15:56.318 (31ms) > Enter [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:905 @ 11/03/26 06:15:56.318 STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 11/03/26 06:15:56.318 END STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 11/03/26 06:15:56.392 (75ms) STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 11/03/26 06:15:56.393 END STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 11/03/26 06:15:56.44 (47ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 11/03/26 06:15:56.44 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 11/03/26 06:15:56.714 (275ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 11/03/26 06:15:56.714 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 11/03/26 06:15:56.775 (61ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 11/03/26 06:15:56.775 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 11/03/26 06:15:57.05 (275ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 11/03/26 06:15:57.05 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 11/03/26 06:15:57.095 (45ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 11/03/26 06:15:57.095 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 11/03/26 06:15:57.16 (65ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 11/03/26 06:15:57.16 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 11/03/26 06:15:57.435 (275ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 11/03/26 06:15:57.435 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 11/03/26 06:15:57.477 (42ms) < Exit [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:905 @ 11/03/26 06:15:57.477 (1.159s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:57.477 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:57.495 (19ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 11/03/26 06:15:57.495 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 11/03/26 06:15:57.511 (15ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:15:57.511 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/03/26 06:15:57.545 (34ms) > Enter [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:917 @ 11/03/26 06:15:57.545 STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 11/03/26 06:15:57.545 END STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 11/03/26 06:15:57.636 (91ms) STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 11/03/26 06:15:57.636 END STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 11/03/26 06:15:57.688 (52ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 11/03/26 06:15:57.688 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 11/03/26 06:15:57.963 (275ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 11/03/26 06:15:57.963 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 11/03/26 06:15:58.034 (71ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 11/03/26 06:15:58.034 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 11/03/26 06:15:58.31 (277ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 11/03/26 06:15:58.311 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 11/03/26 06:15:58.355 (45ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 11/03/26 06:15:58.355 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 11/03/26 06:15:58.417 (62ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 11/03/26 06:15:58.417 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 11/03/26 06:15:58.692 (275ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 11/03/26 06:15:58.692 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 11/03/26 06:15:58.73 (38ms) < Exit [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:917 @ 11/03/26 06:15:58.73 (1.185s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:58.73 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/03/26 06:15:58.75 (20ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 11/03/26 06:15:58.75 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 11/03/26 06:15:58.766 (16ms)