> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 09/19/26 02:50:48.309
< Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 09/19/26 02:50:48.31 (1ms)
[FAILED] Timed out after 600.001s.
The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:112 with:
Expected object to be comparable, diff: []v1.Condition(
- nil,
+ {
+ {
+ Type: "Complete",
+ Status: "True",
+ Reason: "AllJobsCompleted",
+ Message: "jobset completed successfully",
+ },
+ },
)
In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:130 @ 09/19/26 03:00:48.658
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 02:50:48.31
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 02:50:48.355 (45ms)
> Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:79 @ 09/19/26 02:50:48.355
STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:85 @ 09/19/26 02:50:48.355
END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:85 @ 09/19/26 02:50:48.4 (45ms)
STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:90 @ 09/19/26 02:50:48.4
END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:90 @ 09/19/26 02:50:48.657 (257ms)
STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:108 @ 09/19/26 02:50:48.657
END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:108 @ 09/19/26 03:00:48.658 (10m0.001s)
[FAILED] Timed out after 600.001s.
The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:112 with:
Expected object to be comparable, diff: []v1.Condition(
- nil,
+ {
+ {
+ Type: "Complete",
+ Status: "True",
+ Reason: "AllJobsCompleted",
+ Message: "jobset completed successfully",
+ },
+ },
)
In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:130 @ 09/19/26 03:00:48.658
< Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:79 @ 09/19/26 03:00:48.659 (10m0.303s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:00:48.659
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:00:48.663 (5ms)
[FAILED] Expected success, but got an error:
<*errors.StatusError | 0x2f19200aa960>:
admission webhook "validator.trainjob.trainer.kubeflow.org" denied the request: spec.RuntimeRef: Invalid value: {"name":"deepspeed-distributed","apiGroup":"trainer.kubeflow.org","kind":"ClusterTrainingRuntime"}: ClusterTrainingRuntime.trainer.kubeflow.org "deepspeed-distributed" not found: specified clusterTrainingRuntime must be created before the TrainJob is created
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "admission webhook \"validator.trainjob.trainer.kubeflow.org\" denied the request: spec.RuntimeRef: Invalid value: {\"name\":\"deepspeed-distributed\",\"apiGroup\":\"trainer.kubeflow.org\",\"kind\":\"ClusterTrainingRuntime\"}: ClusterTrainingRuntime.trainer.kubeflow.org \"deepspeed-distributed\" not found: specified clusterTrainingRuntime must be created before the TrainJob is created",
Reason: "Forbidden",
Details: nil,
Code: 403,
},
}
In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:148 @ 09/19/26 03:00:48.704
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:00:48.664
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:00:48.671 (8ms)
> Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:137 @ 09/19/26 03:00:48.671
STEP: Create a TrainJob with deepspeed-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:147 @ 09/19/26 03:00:48.671
END STEP: Create a TrainJob with deepspeed-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:147 @ 09/19/26 03:00:48.704 (33ms)
[FAILED] Expected success, but got an error:
<*errors.StatusError | 0x2f19200aa960>:
admission webhook "validator.trainjob.trainer.kubeflow.org" denied the request: spec.RuntimeRef: Invalid value: {"name":"deepspeed-distributed","apiGroup":"trainer.kubeflow.org","kind":"ClusterTrainingRuntime"}: ClusterTrainingRuntime.trainer.kubeflow.org "deepspeed-distributed" not found: specified clusterTrainingRuntime must be created before the TrainJob is created
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "admission webhook \"validator.trainjob.trainer.kubeflow.org\" denied the request: spec.RuntimeRef: Invalid value: {\"name\":\"deepspeed-distributed\",\"apiGroup\":\"trainer.kubeflow.org\",\"kind\":\"ClusterTrainingRuntime\"}: ClusterTrainingRuntime.trainer.kubeflow.org \"deepspeed-distributed\" not found: specified clusterTrainingRuntime must be created before the TrainJob is created",
Reason: "Forbidden",
Details: nil,
Code: 403,
},
}
In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:148 @ 09/19/26 03:00:48.704
< Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:137 @ 09/19/26 03:00:48.704 (33ms)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:00:48.704
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:00:48.713 (9ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:00:48.713
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:00:48.72 (7ms)
> Enter [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:191 @ 09/19/26 03:00:48.72
STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:197 @ 09/19/26 03:00:48.72
END STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:197 @ 09/19/26 03:00:48.732 (13ms)
STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:202 @ 09/19/26 03:00:48.732
END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:202 @ 09/19/26 03:00:48.993 (261ms)
STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:220 @ 09/19/26 03:00:48.993
END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:220 @ 09/19/26 03:02:43.42 (1m54.426s)
< Exit [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:191 @ 09/19/26 03:02:43.42 (1m54.7s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:02:43.42
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:02:43.426 (7ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:02:43.427
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:02:43.434 (7ms)
> Enter [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:249 @ 09/19/26 03:02:43.434
STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:255 @ 09/19/26 03:02:43.434
END STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:255 @ 09/19/26 03:02:43.471 (37ms)
STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:260 @ 09/19/26 03:02:43.471
END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:260 @ 09/19/26 03:02:43.728 (257ms)
STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:278 @ 09/19/26 03:02:43.728
END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:278 @ 09/19/26 03:03:26.09 (42.362s)
< Exit [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:249 @ 09/19/26 03:03:26.09 (42.656s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:03:26.09
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:03:26.094 (4ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:03:26.095
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:03:26.101 (7ms)
> Enter [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:306 @ 09/19/26 03:03:26.101
STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:325 @ 09/19/26 03:03:26.101
END STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:325 @ 09/19/26 03:03:26.122 (20ms)
STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:329 @ 09/19/26 03:03:26.122
END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:329 @ 09/19/26 03:03:26.378 (256ms)
STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:346 @ 09/19/26 03:03:26.378
END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:346 @ 09/19/26 03:03:54.293 (27.916s)
< Exit [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:306 @ 09/19/26 03:03:54.293 (28.192s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:03:54.293
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:03:54.298 (4ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:03:54.298
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:03:54.305 (7ms)
> Enter [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:374 @ 09/19/26 03:03:54.305
STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:426 @ 09/19/26 03:03:54.305
END STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:426 @ 09/19/26 03:03:54.327 (22ms)
STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:430 @ 09/19/26 03:03:54.327
END STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:430 @ 09/19/26 03:03:54.334 (6ms)
< Exit [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:374 @ 09/19/26 03:03:54.334 (28ms)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:03:54.334
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:03:54.34 (6ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:03:54.34
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:03:54.35 (11ms)
> Enter [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:444 @ 09/19/26 03:03:54.35
STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:498 @ 09/19/26 03:03:54.35
END STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:498 @ 09/19/26 03:03:54.366 (15ms)
STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:503 @ 09/19/26 03:03:54.366
END STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:503 @ 09/19/26 03:03:54.371 (5ms)
STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:520 @ 09/19/26 03:03:55.371
END STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:520 @ 09/19/26 03:03:55.389 (19ms)
STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:539 @ 09/19/26 03:03:55.39
END STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:539 @ 09/19/26 03:03:55.392 (3ms)
< Exit [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:444 @ 09/19/26 03:03:55.392 (1.042s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:03:55.392
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:03:55.398 (6ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:03:55.398
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:03:55.405 (7ms)
> Enter [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:561 @ 09/19/26 03:03:55.405
STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:568 @ 09/19/26 03:03:55.405
END STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:568 @ 09/19/26 03:03:55.418 (13ms)
STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:572 @ 09/19/26 03:03:55.418
END STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:572 @ 09/19/26 03:04:10.419 (15.001s)
< Exit [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:561 @ 09/19/26 03:04:10.419 (15.014s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:04:10.419
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:04:10.424 (5ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:04:10.425
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:04:10.432 (7ms)
> Enter [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:584 @ 09/19/26 03:04:10.432
STEP: Waiting for TrainJob to fail due to deadline - /workspace/source/src_code/test/e2e/e2e_test.go:599 @ 09/19/26 03:04:10.445
STEP: Ensuring the underlying JobSet is deleted - /workspace/source/src_code/test/e2e/e2e_test.go:606 @ 09/19/26 03:04:20.085
< Exit [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:584 @ 09/19/26 03:04:20.087 (9.655s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:04:20.087
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:04:20.091 (4ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:04:20.091
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:04:20.098 (7ms)
> Enter [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:837 @ 09/19/26 03:04:20.098
STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:729 @ 09/19/26 03:04:20.125
END STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:729 @ 09/19/26 03:04:20.148 (24ms)
STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/19/26 03:04:20.148
END STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/19/26 03:04:20.158 (10ms)
STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:746 @ 09/19/26 03:04:20.158
END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:746 @ 09/19/26 03:04:20.416 (258ms)
STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:757 @ 09/19/26 03:04:20.416
END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:757 @ 09/19/26 03:04:20.431 (15ms)
STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/19/26 03:04:20.431
END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/19/26 03:04:20.691 (260ms)
STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:779 @ 09/19/26 03:04:20.691
END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:779 @ 09/19/26 03:04:20.702 (11ms)
STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:800 @ 09/19/26 03:04:20.702
END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:800 @ 09/19/26 03:04:20.716 (13ms)
STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:811 @ 09/19/26 03:04:20.716
END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:811 @ 09/19/26 03:04:20.972 (256ms)
STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:823 @ 09/19/26 03:04:20.972
END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:823 @ 09/19/26 03:04:21.234 (262ms)
< Exit [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:837 @ 09/19/26 03:04:21.234 (1.136s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:04:21.234
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:04:21.238 (4ms)
> Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:733 @ 09/19/26 03:04:21.238
< Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:733 @ 09/19/26 03:04:21.242 (4ms)
> Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:04:21.243
< Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/19/26 03:04:21.251 (9ms)
> Enter [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:849 @ 09/19/26 03:04:21.251
STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:729 @ 09/19/26 03:04:21.251
END STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:729 @ 09/19/26 03:04:21.276 (25ms)
STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/19/26 03:04:21.276
END STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/19/26 03:04:21.287 (11ms)
STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:746 @ 09/19/26 03:04:21.287
END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:746 @ 09/19/26 03:04:21.543 (256ms)
STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:757 @ 09/19/26 03:04:21.543
END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:757 @ 09/19/26 03:04:21.556 (13ms)
STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/19/26 03:04:21.556
END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/19/26 03:04:21.813 (256ms)
STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:779 @ 09/19/26 03:04:21.813
END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:779 @ 09/19/26 03:04:21.826 (13ms)
STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:800 @ 09/19/26 03:04:21.826
END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:800 @ 09/19/26 03:04:21.839 (13ms)
STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:811 @ 09/19/26 03:04:21.839
END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:811 @ 09/19/26 03:04:22.097 (258ms)
STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:823 @ 09/19/26 03:04:22.097
END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:823 @ 09/19/26 03:04:22.359 (262ms)
< Exit [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:849 @ 09/19/26 03:04:22.359 (1.108s)
> Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:04:22.359
< Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/19/26 03:04:22.366 (6ms)
> Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:733 @ 09/19/26 03:04:22.366
< Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:733 @ 09/19/26 03:04:22.37 (4ms)