> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 09/19/26 01:01:17.49
< Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 09/19/26 01:01:17.491 (1ms)
[FAILED] Timed out after 600.000s.
The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:112 with:
Expected object to be comparable, diff: []v1.Condition(
- nil,
+ {
+ {
+ Type: "Complete",
+ Status: "True",
+ Reason: "AllJobsCompleted",
+ Message: "jobset completed successfully",
+ },
+ },
)
In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:130 @ 09/19/26 01:11:17.895
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:01:17.491
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:01:17.554 (63ms)
> Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:79 @ 09/19/26 01:01:17.554
STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:85 @ 09/19/26 01:01:17.554
END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:85 @ 09/19/26 01:01:17.636 (82ms)
STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:90 @ 09/19/26 01:01:17.636
END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:90 @ 09/19/26 01:01:17.894 (258ms)
STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:108 @ 09/19/26 01:01:17.894
END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:108 @ 09/19/26 01:11:17.895 (10m0.001s)
[FAILED] Timed out after 600.000s.
The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:112 with:
Expected object to be comparable, diff: []v1.Condition(
- nil,
+ {
+ {
+ Type: "Complete",
+ Status: "True",
+ Reason: "AllJobsCompleted",
+ Message: "jobset completed successfully",
+ },
+ },
)
In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:130 @ 09/19/26 01:11:17.895
< Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:79 @ 09/19/26 01:11:17.895 (10m0.342s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:11:17.895
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:11:17.901 (6ms)
[FAILED] Expected success, but got an error:
<*errors.StatusError | 0x215b05000820>:
admission webhook "validator.trainjob.trainer.kubeflow.org" denied the request: spec.RuntimeRef: Invalid value: {"name":"deepspeed-distributed","apiGroup":"trainer.kubeflow.org","kind":"ClusterTrainingRuntime"}: ClusterTrainingRuntime.trainer.kubeflow.org "deepspeed-distributed" not found: specified clusterTrainingRuntime must be created before the TrainJob is created
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "admission webhook \"validator.trainjob.trainer.kubeflow.org\" denied the request: spec.RuntimeRef: Invalid value: {\"name\":\"deepspeed-distributed\",\"apiGroup\":\"trainer.kubeflow.org\",\"kind\":\"ClusterTrainingRuntime\"}: ClusterTrainingRuntime.trainer.kubeflow.org \"deepspeed-distributed\" not found: specified clusterTrainingRuntime must be created before the TrainJob is created",
Reason: "Forbidden",
Details: nil,
Code: 403,
},
}
In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:148 @ 09/19/26 01:11:17.945
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:11:17.901
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:11:17.909 (8ms)
> Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:137 @ 09/19/26 01:11:17.909
STEP: Create a TrainJob with deepspeed-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:147 @ 09/19/26 01:11:17.909
END STEP: Create a TrainJob with deepspeed-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:147 @ 09/19/26 01:11:17.945 (35ms)
[FAILED] Expected success, but got an error:
<*errors.StatusError | 0x215b05000820>:
admission webhook "validator.trainjob.trainer.kubeflow.org" denied the request: spec.RuntimeRef: Invalid value: {"name":"deepspeed-distributed","apiGroup":"trainer.kubeflow.org","kind":"ClusterTrainingRuntime"}: ClusterTrainingRuntime.trainer.kubeflow.org "deepspeed-distributed" not found: specified clusterTrainingRuntime must be created before the TrainJob is created
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "admission webhook \"validator.trainjob.trainer.kubeflow.org\" denied the request: spec.RuntimeRef: Invalid value: {\"name\":\"deepspeed-distributed\",\"apiGroup\":\"trainer.kubeflow.org\",\"kind\":\"ClusterTrainingRuntime\"}: ClusterTrainingRuntime.trainer.kubeflow.org \"deepspeed-distributed\" not found: specified clusterTrainingRuntime must be created before the TrainJob is created",
Reason: "Forbidden",
Details: nil,
Code: 403,
},
}
In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:148 @ 09/19/26 01:11:17.945
< Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:137 @ 09/19/26 01:11:17.945 (36ms)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:11:17.945
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:11:17.951 (7ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:11:17.951
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:11:17.96 (8ms)
> Enter [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:191 @ 09/19/26 01:11:17.96
STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:197 @ 09/19/26 01:11:17.96
END STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:197 @ 09/19/26 01:11:17.975 (15ms)
STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:202 @ 09/19/26 01:11:17.975
END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:202 @ 09/19/26 01:11:18.235 (259ms)
STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:220 @ 09/19/26 01:11:18.235
END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:220 @ 09/19/26 01:13:14.056 (1m55.821s)
< Exit [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:191 @ 09/19/26 01:13:14.056 (1m56.096s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:13:14.056
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:13:14.061 (5ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:13:14.061
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:13:14.071 (9ms)
> Enter [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:249 @ 09/19/26 01:13:14.071
STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:255 @ 09/19/26 01:13:14.071
END STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:255 @ 09/19/26 01:13:14.111 (40ms)
STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:260 @ 09/19/26 01:13:14.111
END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:260 @ 09/19/26 01:13:14.368 (257ms)
STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:278 @ 09/19/26 01:13:14.368
END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:278 @ 09/19/26 01:13:59.145 (44.776s)
< Exit [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:249 @ 09/19/26 01:13:59.145 (45.074s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:13:59.145
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:13:59.152 (7ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:13:59.152
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:13:59.16 (7ms)
> Enter [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:306 @ 09/19/26 01:13:59.16
STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:325 @ 09/19/26 01:13:59.16
END STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:325 @ 09/19/26 01:13:59.179 (20ms)
STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:329 @ 09/19/26 01:13:59.179
END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:329 @ 09/19/26 01:13:59.438 (258ms)
STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:346 @ 09/19/26 01:13:59.438
END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:346 @ 09/19/26 01:14:23.882 (24.444s)
< Exit [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:306 @ 09/19/26 01:14:23.882 (24.723s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:23.882
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:23.89 (8ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:14:23.891
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:14:23.91 (19ms)
> Enter [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:374 @ 09/19/26 01:14:23.91
STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:426 @ 09/19/26 01:14:23.91
END STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:426 @ 09/19/26 01:14:23.931 (21ms)
STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:430 @ 09/19/26 01:14:23.931
END STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:430 @ 09/19/26 01:14:23.939 (7ms)
< Exit [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:374 @ 09/19/26 01:14:23.939 (29ms)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:23.939
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:23.95 (12ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:14:23.951
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:14:23.966 (15ms)
> Enter [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:444 @ 09/19/26 01:14:23.966
STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:498 @ 09/19/26 01:14:23.966
END STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:498 @ 09/19/26 01:14:23.988 (22ms)
STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:503 @ 09/19/26 01:14:23.988
END STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:503 @ 09/19/26 01:14:23.998 (10ms)
STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:520 @ 09/19/26 01:14:24.998
END STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:520 @ 09/19/26 01:14:25.024 (26ms)
STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:539 @ 09/19/26 01:14:25.024
END STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:539 @ 09/19/26 01:14:25.03 (6ms)
< Exit [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:444 @ 09/19/26 01:14:25.03 (1.064s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:25.03
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:25.035 (5ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:14:25.035
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:14:25.044 (9ms)
> Enter [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:561 @ 09/19/26 01:14:25.044
STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:568 @ 09/19/26 01:14:25.044
END STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:568 @ 09/19/26 01:14:25.062 (17ms)
STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:572 @ 09/19/26 01:14:25.062
END STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:572 @ 09/19/26 01:14:40.063 (15.001s)
< Exit [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:561 @ 09/19/26 01:14:40.063 (15.018s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:40.063
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:40.068 (5ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:14:40.069
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:14:40.076 (8ms)
> Enter [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:584 @ 09/19/26 01:14:40.076
STEP: Waiting for TrainJob to fail due to deadline - /workspace/source/src_code/test/e2e/e2e_test.go:599 @ 09/19/26 01:14:40.093
STEP: Ensuring the underlying JobSet is deleted - /workspace/source/src_code/test/e2e/e2e_test.go:606 @ 09/19/26 01:14:50.271
< Exit [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:584 @ 09/19/26 01:14:50.274 (10.197s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:50.274
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:50.28 (7ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:14:50.281
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:14:50.289 (8ms)
> Enter [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:837 @ 09/19/26 01:14:50.289
STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:729 @ 09/19/26 01:14:50.289
END STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:729 @ 09/19/26 01:14:50.316 (28ms)
STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/19/26 01:14:50.316
END STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/19/26 01:14:50.334 (17ms)
STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:746 @ 09/19/26 01:14:50.334
END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:746 @ 09/19/26 01:14:50.59 (257ms)
STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:757 @ 09/19/26 01:14:50.59
END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:757 @ 09/19/26 01:14:50.608 (17ms)
STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/19/26 01:14:50.608
END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/19/26 01:14:50.864 (257ms)
STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:779 @ 09/19/26 01:14:50.864
END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:779 @ 09/19/26 01:14:50.878 (14ms)
STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:800 @ 09/19/26 01:14:50.878
END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:800 @ 09/19/26 01:14:50.895 (17ms)
STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:811 @ 09/19/26 01:14:50.895
END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:811 @ 09/19/26 01:14:51.152 (257ms)
STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:823 @ 09/19/26 01:14:51.152
END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:823 @ 09/19/26 01:14:51.416 (265ms)
< Exit [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:837 @ 09/19/26 01:14:51.416 (1.128s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:51.416
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:51.422 (5ms)
> Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:733 @ 09/19/26 01:14:51.422
< Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:733 @ 09/19/26 01:14:51.429 (7ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:14:51.429
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 01:14:51.437 (8ms)
> Enter [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:849 @ 09/19/26 01:14:51.437
STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:729 @ 09/19/26 01:14:51.437
END STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:729 @ 09/19/26 01:14:51.462 (25ms)
STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/19/26 01:14:51.462
END STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/19/26 01:14:51.473 (11ms)
STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:746 @ 09/19/26 01:14:51.473
END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:746 @ 09/19/26 01:14:51.73 (257ms)
STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:757 @ 09/19/26 01:14:51.73
END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:757 @ 09/19/26 01:14:51.746 (16ms)
STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/19/26 01:14:51.746
END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/19/26 01:14:52.003 (257ms)
STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:779 @ 09/19/26 01:14:52.003
END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:779 @ 09/19/26 01:14:52.015 (12ms)
STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:800 @ 09/19/26 01:14:52.015
END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:800 @ 09/19/26 01:14:52.033 (17ms)
STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:811 @ 09/19/26 01:14:52.033
END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:811 @ 09/19/26 01:14:52.29 (257ms)
STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:823 @ 09/19/26 01:14:52.29
END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:823 @ 09/19/26 01:14:52.555 (265ms)
< Exit [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:849 @ 09/19/26 01:14:52.555 (1.118s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:52.555
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 01:14:52.562 (8ms)
> Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:733 @ 09/19/26 01:14:52.562
< Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:733 @ 09/19/26 01:14:52.574 (12ms)