> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 11/02/26 22:35:05.834 < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 11/02/26 22:35:05.834 (1ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:35:05.835 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:35:05.981 (146ms) > Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:84 @ 11/02/26 22:35:05.981 STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:93 @ 11/02/26 22:35:05.981 END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:93 @ 11/02/26 22:35:06.329 (348ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:98 @ 11/02/26 22:35:06.329 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:98 @ 11/02/26 22:35:07.41 (1.081s) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:116 @ 11/02/26 22:35:07.41 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:116 @ 11/02/26 22:39:21.788 (4m14.378s) < Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:84 @ 11/02/26 22:39:21.788 (4m15.807s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:39:21.788 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:39:21.807 (19ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:39:21.807 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:39:21.84 (32ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:145 @ 11/02/26 22:39:21.84 [SKIPPED] MPI runtime image is not available In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:147 @ 11/02/26 22:39:21.84 < Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:145 @ 11/02/26 22:39:21.84 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:39:21.84 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:39:21.858 (18ms) [FAILED] Expected success, but got an error: <*errors.StatusError | 0x1b5095914500>: admission webhook "validator.trainjob.trainer.kubeflow.org" denied the request: spec.runtimeRef: Invalid value: {"name":"deepspeed-distributed","apiGroup":"trainer.kubeflow.org","kind":"ClusterTrainingRuntime"}: ClusterTrainingRuntime.trainer.kubeflow.org "deepspeed-distributed" not found: specified clusterTrainingRuntime must be created before the TrainJob is created { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "admission webhook \"validator.trainjob.trainer.kubeflow.org\" denied the request: spec.runtimeRef: Invalid value: {\"name\":\"deepspeed-distributed\",\"apiGroup\":\"trainer.kubeflow.org\",\"kind\":\"ClusterTrainingRuntime\"}: ClusterTrainingRuntime.trainer.kubeflow.org \"deepspeed-distributed\" not found: specified clusterTrainingRuntime must be created before the TrainJob is created", Reason: "Forbidden", Details: nil, Code: 403, }, } In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:216 @ 11/02/26 22:39:22.224 > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:39:21.858 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:39:21.891 (33ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference and OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:203 @ 11/02/26 22:39:21.891 STEP: Create a TrainJob with deepspeed-distributed runtime reference and the OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:215 @ 11/02/26 22:39:21.891 END STEP: Create a TrainJob with deepspeed-distributed runtime reference and the OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:215 @ 11/02/26 22:39:22.224 (333ms) [FAILED] Expected success, but got an error: <*errors.StatusError | 0x1b5095914500>: admission webhook "validator.trainjob.trainer.kubeflow.org" denied the request: spec.runtimeRef: Invalid value: {"name":"deepspeed-distributed","apiGroup":"trainer.kubeflow.org","kind":"ClusterTrainingRuntime"}: ClusterTrainingRuntime.trainer.kubeflow.org "deepspeed-distributed" not found: specified clusterTrainingRuntime must be created before the TrainJob is created { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "admission webhook \"validator.trainjob.trainer.kubeflow.org\" denied the request: spec.runtimeRef: Invalid value: {\"name\":\"deepspeed-distributed\",\"apiGroup\":\"trainer.kubeflow.org\",\"kind\":\"ClusterTrainingRuntime\"}: ClusterTrainingRuntime.trainer.kubeflow.org \"deepspeed-distributed\" not found: specified clusterTrainingRuntime must be created before the TrainJob is created", Reason: "Forbidden", Details: nil, Code: 403, }, } In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:216 @ 11/02/26 22:39:22.224 < Exit [It] should create TrainJob with DeepSpeed runtime reference and OpenMPI 4.0 image - /workspace/source/src_code/test/e2e/e2e_test.go:203 @ 11/02/26 22:39:22.224 (333ms) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:39:22.224 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:39:22.241 (17ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:39:22.242 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:39:22.274 (33ms) > Enter [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:259 @ 11/02/26 22:39:22.274 STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:265 @ 11/02/26 22:39:22.275 END STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:265 @ 11/02/26 22:39:22.393 (118ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:270 @ 11/02/26 22:39:22.393 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:270 @ 11/02/26 22:39:23.743 (1.35s) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:288 @ 11/02/26 22:39:23.743 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:288 @ 11/02/26 22:41:42.496 (2m18.752s) < Exit [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:259 @ 11/02/26 22:41:42.496 (2m20.221s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:41:42.496 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:41:42.513 (18ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:41:42.514 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:41:42.549 (35ms) > Enter [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:317 @ 11/02/26 22:41:42.549 STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:323 @ 11/02/26 22:41:42.549 END STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:323 @ 11/02/26 22:41:42.91 (361ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:328 @ 11/02/26 22:41:42.91 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:328 @ 11/02/26 22:41:43.991 (1.081s) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:346 @ 11/02/26 22:41:43.991 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:346 @ 11/02/26 22:42:51.93 (1m7.939s) < Exit [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:317 @ 11/02/26 22:42:51.93 (1m9.381s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:42:51.93 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:42:51.947 (17ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:42:51.948 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:42:51.982 (34ms) > Enter [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:374 @ 11/02/26 22:42:51.982 STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:393 @ 11/02/26 22:42:51.982 END STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:393 @ 11/02/26 22:42:52.1 (118ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:397 @ 11/02/26 22:42:52.1 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:397 @ 11/02/26 22:42:53.18 (1.08s) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:414 @ 11/02/26 22:42:53.18 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:414 @ 11/02/26 22:43:20.633 (27.452s) < Exit [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:374 @ 11/02/26 22:43:20.633 (28.651s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:20.633 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:20.649 (17ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:43:20.649 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:43:20.687 (37ms) > Enter [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:442 @ 11/02/26 22:43:20.687 STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:494 @ 11/02/26 22:43:20.687 END STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:494 @ 11/02/26 22:43:20.811 (124ms) STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:498 @ 11/02/26 22:43:20.811 END STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:498 @ 11/02/26 22:43:20.827 (16ms) < Exit [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:442 @ 11/02/26 22:43:20.827 (140ms) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:20.827 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:20.844 (17ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:43:20.844 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:43:20.878 (34ms) > Enter [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:512 @ 11/02/26 22:43:20.878 STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:566 @ 11/02/26 22:43:20.878 END STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:566 @ 11/02/26 22:43:21 (122ms) STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:571 @ 11/02/26 22:43:21 END STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:571 @ 11/02/26 22:43:21.016 (15ms) STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:588 @ 11/02/26 22:43:22.016 END STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:588 @ 11/02/26 22:43:22.157 (141ms) STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:607 @ 11/02/26 22:43:22.157 END STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:607 @ 11/02/26 22:43:22.172 (15ms) < Exit [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:512 @ 11/02/26 22:43:22.172 (1.295s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:22.172 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:22.189 (17ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:43:22.189 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:43:22.223 (33ms) > Enter [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:629 @ 11/02/26 22:43:22.223 STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:636 @ 11/02/26 22:43:22.223 END STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:636 @ 11/02/26 22:43:22.341 (118ms) STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:640 @ 11/02/26 22:43:22.341 END STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:640 @ 11/02/26 22:43:37.342 (15.001s) < Exit [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:629 @ 11/02/26 22:43:37.342 (15.119s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:37.342 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:37.361 (19ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:43:37.361 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:43:37.393 (32ms) > Enter [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:652 @ 11/02/26 22:43:37.393 STEP: Waiting for TrainJob to fail due to deadline - /workspace/source/src_code/test/e2e/e2e_test.go:667 @ 11/02/26 22:43:37.514 STEP: Ensuring the underlying JobSet is deleted - /workspace/source/src_code/test/e2e/e2e_test.go:674 @ 11/02/26 22:43:47.38 < Exit [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:652 @ 11/02/26 22:43:47.395 (10.002s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:47.395 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:47.412 (17ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:43:47.412 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:43:47.447 (35ms) > Enter [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:905 @ 11/02/26 22:43:47.447 STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 11/02/26 22:43:47.447 END STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 11/02/26 22:43:47.629 (182ms) STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 11/02/26 22:43:47.63 END STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 11/02/26 22:43:47.75 (120ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 11/02/26 22:43:47.75 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 11/02/26 22:43:48.564 (814ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 11/02/26 22:43:48.564 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 11/02/26 22:43:48.701 (137ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 11/02/26 22:43:48.701 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 11/02/26 22:43:49.249 (548ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 11/02/26 22:43:49.249 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 11/02/26 22:43:49.336 (87ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 11/02/26 22:43:49.336 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 11/02/26 22:43:49.474 (138ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 11/02/26 22:43:49.474 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 11/02/26 22:43:50.023 (549ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 11/02/26 22:43:50.023 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 11/02/26 22:43:50.073 (50ms) < Exit [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:905 @ 11/02/26 22:43:50.073 (2.626s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:50.073 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:50.092 (19ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 11/02/26 22:43:50.092 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 11/02/26 22:43:50.109 (17ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:43:50.109 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:61 @ 11/02/26 22:43:50.145 (35ms) > Enter [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:917 @ 11/02/26 22:43:50.145 STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 11/02/26 22:43:50.145 END STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:797 @ 11/02/26 22:43:50.317 (172ms) STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 11/02/26 22:43:50.317 END STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:810 @ 11/02/26 22:43:50.436 (118ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 11/02/26 22:43:50.436 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:814 @ 11/02/26 22:43:51.249 (813ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 11/02/26 22:43:51.249 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:825 @ 11/02/26 22:43:51.384 (135ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 11/02/26 22:43:51.384 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:834 @ 11/02/26 22:43:51.931 (547ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 11/02/26 22:43:51.931 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:847 @ 11/02/26 22:43:52.015 (83ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 11/02/26 22:43:52.015 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:868 @ 11/02/26 22:43:52.149 (135ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 11/02/26 22:43:52.149 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:879 @ 11/02/26 22:43:52.964 (815ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 11/02/26 22:43:52.964 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:891 @ 11/02/26 22:43:53.012 (48ms) < Exit [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:917 @ 11/02/26 22:43:53.012 (2.867s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:53.012 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:76 @ 11/02/26 22:43:53.045 (33ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 11/02/26 22:43:53.045 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:801 @ 11/02/26 22:43:53.071 (26ms)