> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:42 @ 10/22/26 17:07:23.708 < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:42 @ 10/22/26 17:07:23.709 (1ms) [FAILED] Timed out after 600.001s. The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:84 with: Expected object to be comparable, diff: []v1.Condition( - nil, + { + { + Type: "Complete", + Status: "True", + Reason: "AllJobsCompleted", + Message: "jobset completed successfully", + }, + }, ) In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:102 @ 10/22/26 17:17:24.453 > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 10/22/26 17:07:23.709 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 10/22/26 17:07:23.834 (125ms) > Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:51 @ 10/22/26 17:07:23.834 STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:57 @ 10/22/26 17:07:23.834 END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:57 @ 10/22/26 17:07:23.915 (81ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:62 @ 10/22/26 17:07:23.915 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:62 @ 10/22/26 17:07:24.452 (537ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:80 @ 10/22/26 17:07:24.452 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:80 @ 10/22/26 17:17:24.453 (10m0.001s) [FAILED] Timed out after 600.001s. The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:84 with: Expected object to be comparable, diff: []v1.Condition( - nil, + { + { + Type: "Complete", + Status: "True", + Reason: "AllJobsCompleted", + Message: "jobset completed successfully", + }, + }, ) In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:102 @ 10/22/26 17:17:24.453 < Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:51 @ 10/22/26 17:17:24.454 (10m0.619s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 10/22/26 17:17:24.454 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 10/22/26 17:17:24.467 (14ms) [FAILED] Timed out after 600.000s. The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:124 with: Expected object to be comparable, diff: []v1alpha1.JobStatus(Inverse(cmpopts.SortSlices, []v1alpha1.JobStatus{ { ... // 2 identical fields Succeeded: &0, Failed: &0, - Active: &0, + Active: &1, Suspended: &0, }, { ... // 2 identical fields Succeeded: &0, Failed: &0, - Active: &0, + Active: &1, Suspended: &0, }, })) In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:142 @ 10/22/26 17:27:24.576 > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 10/22/26 17:17:24.467 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 10/22/26 17:17:24.495 (28ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:109 @ 10/22/26 17:17:24.495 STEP: Create a TrainJob with deepspeed-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:115 @ 10/22/26 17:17:24.495 END STEP: Create a TrainJob with deepspeed-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:115 @ 10/22/26 17:17:24.575 (80ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:120 @ 10/22/26 17:17:24.575 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:120 @ 10/22/26 17:27:24.576 (10m0.001s) [FAILED] Timed out after 600.000s. The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:124 with: Expected object to be comparable, diff: []v1alpha1.JobStatus(Inverse(cmpopts.SortSlices, []v1alpha1.JobStatus{ { ... // 2 identical fields Succeeded: &0, Failed: &0, - Active: &0, + Active: &1, Suspended: &0, }, { ... // 2 identical fields Succeeded: &0, Failed: &0, - Active: &0, + Active: &1, Suspended: &0, }, })) In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:142 @ 10/22/26 17:27:24.576 < Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:109 @ 10/22/26 17:27:24.576 (10m0.081s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 10/22/26 17:27:24.576 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 10/22/26 17:27:24.592 (16ms)