> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 09/18/26 13:25:29.32 < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 09/18/26 13:25:29.321 (1ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:25:29.322 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:25:29.372 (51ms) > Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:79 @ 09/18/26 13:25:29.372 STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:88 @ 09/18/26 13:25:29.372 END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:88 @ 09/18/26 13:25:29.422 (49ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:93 @ 09/18/26 13:25:29.422 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:93 @ 09/18/26 13:25:29.679 (257ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:111 @ 09/18/26 13:25:29.679 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:111 @ 09/18/26 13:29:53.682 (4m24.003s) < Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:79 @ 09/18/26 13:29:53.682 (4m24.309s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:29:53.682 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:29:53.689 (7ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:29:53.689 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:29:53.697 (9ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:140 @ 09/18/26 13:29:53.697 [SKIPPED] MPI runtime image is not available In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:142 @ 09/18/26 13:29:53.698 < Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:140 @ 09/18/26 13:29:53.698 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:29:53.698 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:29:53.705 (7ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:29:53.705 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:29:53.718 (13ms) > Enter [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:197 @ 09/18/26 13:29:53.718 STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:203 @ 09/18/26 13:29:53.718 END STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:203 @ 09/18/26 13:29:53.769 (51ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:208 @ 09/18/26 13:29:53.769 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:208 @ 09/18/26 13:29:54.027 (258ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:226 @ 09/18/26 13:29:54.027 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:226 @ 09/18/26 13:32:14.448 (2m20.422s) < Exit [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:197 @ 09/18/26 13:32:14.448 (2m20.73s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:32:14.448 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:32:14.453 (5ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:32:14.454 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:32:14.463 (9ms) > Enter [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:255 @ 09/18/26 13:32:14.463 STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:261 @ 09/18/26 13:32:14.463 END STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:261 @ 09/18/26 13:32:14.505 (42ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:266 @ 09/18/26 13:32:14.505 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:266 @ 09/18/26 13:32:14.761 (256ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:284 @ 09/18/26 13:32:14.761 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:284 @ 09/18/26 13:32:58.808 (44.047s) < Exit [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:255 @ 09/18/26 13:32:58.808 (44.345s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:32:58.808 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:32:58.813 (5ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:32:58.814 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:32:58.823 (9ms) > Enter [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:312 @ 09/18/26 13:32:58.823 STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:331 @ 09/18/26 13:32:58.823 END STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:331 @ 09/18/26 13:32:58.842 (20ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:335 @ 09/18/26 13:32:58.842 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:335 @ 09/18/26 13:32:59.101 (258ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:352 @ 09/18/26 13:32:59.101 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:352 @ 09/18/26 13:33:28.963 (29.863s) < Exit [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:312 @ 09/18/26 13:33:28.963 (30.141s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:28.963 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:28.973 (10ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:28.973 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:28.983 (10ms) > Enter [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:380 @ 09/18/26 13:33:28.983 STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:432 @ 09/18/26 13:33:28.983 END STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:432 @ 09/18/26 13:33:29.013 (30ms) STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:436 @ 09/18/26 13:33:29.013 END STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:436 @ 09/18/26 13:33:29.018 (5ms) < Exit [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:380 @ 09/18/26 13:33:29.018 (35ms) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:29.018 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:29.028 (10ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:29.028 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:29.043 (15ms) > Enter [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:450 @ 09/18/26 13:33:29.043 STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:504 @ 09/18/26 13:33:29.043 END STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:504 @ 09/18/26 13:33:29.062 (19ms) STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:509 @ 09/18/26 13:33:29.062 END STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:509 @ 09/18/26 13:33:29.077 (15ms) STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:526 @ 09/18/26 13:33:30.077 END STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:526 @ 09/18/26 13:33:30.101 (24ms) STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:545 @ 09/18/26 13:33:30.101 END STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:545 @ 09/18/26 13:33:30.104 (3ms) < Exit [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:450 @ 09/18/26 13:33:30.104 (1.061s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:30.104 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:30.109 (5ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:30.109 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:30.12 (10ms) > Enter [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:567 @ 09/18/26 13:33:30.12 STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:574 @ 09/18/26 13:33:30.12 END STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:574 @ 09/18/26 13:33:30.137 (17ms) STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:578 @ 09/18/26 13:33:30.137 END STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:578 @ 09/18/26 13:33:45.137 (15.001s) < Exit [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:567 @ 09/18/26 13:33:45.137 (15.017s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:45.137 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:45.145 (8ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:45.145 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:45.158 (13ms) > Enter [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:590 @ 09/18/26 13:33:45.158 STEP: Waiting for TrainJob to fail due to deadline - /workspace/source/src_code/test/e2e/e2e_test.go:605 @ 09/18/26 13:33:45.185 STEP: Ensuring the underlying JobSet is deleted - /workspace/source/src_code/test/e2e/e2e_test.go:612 @ 09/18/26 13:33:55.114 < Exit [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:590 @ 09/18/26 13:33:55.117 (9.959s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:55.117 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:55.128 (11ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:55.128 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:55.138 (10ms) > Enter [It] should inject runtime configuration which allows the runtime status endpoint to be called - /workspace/source/src_code/test/e2e/e2e_test.go:620 @ 09/18/26 13:33:55.138 [SKIPPED] TrainJobStatus feature gate is not enabled In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:622 @ 09/18/26 13:33:55.138 < Exit [It] should inject runtime configuration which allows the runtime status endpoint to be called - /workspace/source/src_code/test/e2e/e2e_test.go:620 @ 09/18/26 13:33:55.138 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:55.138 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:55.148 (10ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:55.148 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:55.168 (19ms) > Enter [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:846 @ 09/18/26 13:33:55.168 STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:738 @ 09/18/26 13:33:55.168 END STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:738 @ 09/18/26 13:33:55.193 (25ms) STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:751 @ 09/18/26 13:33:55.193 END STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:751 @ 09/18/26 13:33:55.203 (10ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:755 @ 09/18/26 13:33:55.203 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:755 @ 09/18/26 13:33:55.46 (257ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/18/26 13:33:55.46 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/18/26 13:33:55.477 (17ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:775 @ 09/18/26 13:33:55.477 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:775 @ 09/18/26 13:33:55.733 (257ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:788 @ 09/18/26 13:33:55.733 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:788 @ 09/18/26 13:33:55.746 (12ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:809 @ 09/18/26 13:33:55.746 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:809 @ 09/18/26 13:33:55.762 (16ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:820 @ 09/18/26 13:33:55.762 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:820 @ 09/18/26 13:33:56.018 (256ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:832 @ 09/18/26 13:33:56.018 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:832 @ 09/18/26 13:33:56.285 (267ms) < Exit [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:846 @ 09/18/26 13:33:56.285 (1.118s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:56.285 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:56.292 (7ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/18/26 13:33:56.292 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/18/26 13:33:56.298 (6ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:56.298 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:56.311 (13ms) > Enter [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:858 @ 09/18/26 13:33:56.311 STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:738 @ 09/18/26 13:33:56.311 END STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:738 @ 09/18/26 13:33:56.336 (25ms) STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:751 @ 09/18/26 13:33:56.336 END STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:751 @ 09/18/26 13:33:56.354 (18ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:755 @ 09/18/26 13:33:56.354 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:755 @ 09/18/26 13:33:56.611 (257ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/18/26 13:33:56.611 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/18/26 13:33:56.628 (17ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:775 @ 09/18/26 13:33:56.628 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:775 @ 09/18/26 13:33:56.885 (257ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:788 @ 09/18/26 13:33:56.885 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:788 @ 09/18/26 13:33:56.898 (13ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:809 @ 09/18/26 13:33:56.898 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:809 @ 09/18/26 13:33:56.918 (20ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:820 @ 09/18/26 13:33:56.918 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:820 @ 09/18/26 13:33:57.175 (258ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:832 @ 09/18/26 13:33:57.176 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:832 @ 09/18/26 13:33:57.44 (264ms) < Exit [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:858 @ 09/18/26 13:33:57.44 (1.129s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:57.44 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:57.447 (7ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/18/26 13:33:57.447 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/18/26 13:33:57.453 (6ms) > Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:45 @ 09/18/26 13:33:57.473 Created shared test namespace: rhai-e2e-progression-sjz9b < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:45 @ 09/18/26 13:33:57.518 (46ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:33:57.519 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:33:57.544 (25ms) > Enter [It] should poll metrics and update trainerStatus annotation during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:150 @ 09/18/26 13:33:57.544 STEP: Creating TrainJob with progression tracking enabled - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:172 @ 09/18/26 13:33:57.544 STEP: Waiting for TrainJob pod to be running with IP - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:175 @ 09/18/26 13:33:57.565 STEP: Waiting for trainerStatus annotation to be populated - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:195 @ 09/18/26 13:34:59.724 STEP: Verifying trainerStatus is continuously updated during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:222 @ 09/18/26 13:34:59.729 STEP: Waiting for TrainJob to complete and verify final status - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:241 @ 09/18/26 13:34:59.732 < Exit [It] should poll metrics and update trainerStatus annotation during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:150 @ 09/18/26 13:35:23.803 (1m26.259s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:35:23.803 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:35:23.81 (6ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:35:23.81 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:35:23.824 (14ms) > Enter [It] should NOT create trainerStatus annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:274 @ 09/18/26 13:35:23.824 STEP: Creating TrainJob without progression tracking annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:289 @ 09/18/26 13:35:23.824 STEP: Verifying trainerStatus annotation is NOT created (checking over time) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:292 @ 09/18/26 13:35:23.838 < Exit [It] should NOT create trainerStatus annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:274 @ 09/18/26 13:35:58.853 (35.03s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:35:58.853 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:35:58.86 (7ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:35:58.861 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:35:58.877 (16ms) > Enter [It] should NOT enable progression tracking for non-'true' values - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:307 @ 09/18/26 13:35:58.877 STEP: Creating TrainJob with invalid annotation value - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:324 @ 09/18/26 13:35:58.877 STEP: Verifying progression tracking is NOT enabled (checking over time) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:327 @ 09/18/26 13:35:58.891 < Exit [It] should NOT enable progression tracking for non-'true' values - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:307 @ 09/18/26 13:36:13.896 (15.02s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:13.896 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:13.903 (7ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:13.903 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:13.916 (14ms) > Enter [It] should honor custom metrics port and poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:342 @ 09/18/26 13:36:13.916 STEP: Creating TrainJob with custom metrics configuration - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:360 @ 09/18/26 13:36:13.917 STEP: Verifying custom configuration is applied - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:363 @ 09/18/26 13:36:13.931 STEP: Verifying metrics are polled on the custom port - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:370 @ 09/18/26 13:36:13.934 < Exit [It] should honor custom metrics port and poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:342 @ 09/18/26 13:36:17.95 (4.033s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:17.95 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:17.955 (5ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:17.955 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:17.97 (14ms) > Enter [It] should handle minimum SDK-recommended poll interval (5s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:385 @ 09/18/26 13:36:17.97 STEP: Creating TrainJob with minimum poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:403 @ 09/18/26 13:36:17.97 STEP: Verifying minimum interval is respected - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:406 @ 09/18/26 13:36:17.983 < Exit [It] should handle minimum SDK-recommended poll interval (5s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:385 @ 09/18/26 13:36:17.986 (16ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:17.986 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:17.993 (7ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:17.993 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:18.004 (11ms) > Enter [It] should handle maximum SDK-recommended poll interval (300s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:412 @ 09/18/26 13:36:18.004 STEP: Creating TrainJob with maximum poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:430 @ 09/18/26 13:36:18.004 STEP: Verifying maximum interval is respected - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:433 @ 09/18/26 13:36:18.018 < Exit [It] should handle maximum SDK-recommended poll interval (300s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:412 @ 09/18/26 13:36:18.027 (23ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:18.027 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:18.032 (5ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:18.032 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:18.043 (11ms) > Enter [It] should use default interval when annotation is missing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:439 @ 09/18/26 13:36:18.043 STEP: Creating TrainJob without poll interval annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:457 @ 09/18/26 13:36:18.043 STEP: Verifying default interval (30s) is used - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:460 @ 09/18/26 13:36:18.059 < Exit [It] should use default interval when annotation is missing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:439 @ 09/18/26 13:36:18.063 (20ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:18.063 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:18.075 (12ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:18.075 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:18.088 (13ms) > Enter [BeforeEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:470 @ 09/18/26 13:36:18.088 < Exit [BeforeEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:470 @ 09/18/26 13:36:18.1 (12ms) > Enter [It] should capture final status even when job fails - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:487 @ 09/18/26 13:36:18.1 STEP: Creating TrainJob that will fail mid-training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:505 @ 09/18/26 13:36:18.1 STEP: Waiting for trainerStatus annotation to be populated during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:508 @ 09/18/26 13:36:18.112 STEP: Waiting for TrainJob to fail - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:516 @ 09/18/26 13:37:18.271 STEP: Verifying final status is captured with progress=100% even on failure - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:530 @ 09/18/26 13:37:28.304 < Exit [It] should capture final status even when job fails - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:487 @ 09/18/26 13:37:28.307 (1m10.208s) > Enter [AfterEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:481 @ 09/18/26 13:37:28.307 < Exit [AfterEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:481 @ 09/18/26 13:37:28.313 (5ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:37:28.313 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:37:28.318 (5ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:37:28.318 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:37:28.334 (16ms) > Enter [BeforeEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:550 @ 09/18/26 13:37:28.334 < Exit [BeforeEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:550 @ 09/18/26 13:37:28.349 (15ms) > Enter [It] should handle connection errors gracefully without crashing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:567 @ 09/18/26 13:37:28.349 STEP: Creating TrainJob without metrics endpoint - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:585 @ 09/18/26 13:37:28.349 STEP: Waiting for TrainJob pod to be running - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:588 @ 09/18/26 13:37:28.361 STEP: Check controller still reconciles the TrainJob by ensuring it gets marked complete - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:598 @ 09/18/26 13:37:30.473 < Exit [It] should handle connection errors gracefully without crashing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:567 @ 09/18/26 13:37:38.496 (10.147s) > Enter [AfterEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:561 @ 09/18/26 13:37:38.496 < Exit [AfterEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:561 @ 09/18/26 13:37:38.501 (5ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:37:38.501 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:37:38.506 (5ms) > Enter [AfterSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:71 @ 09/18/26 13:37:38.507 Cleaning up test namespace: rhai-e2e-progression-sjz9b < Exit [AfterSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:71 @ 09/18/26 13:37:38.514 (7ms)