> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 09/18/26 13:26:11.089 < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 09/18/26 13:26:11.09 (1ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:26:11.09 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:26:11.143 (52ms) > Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:79 @ 09/18/26 13:26:11.143 STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:88 @ 09/18/26 13:26:11.143 END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:88 @ 09/18/26 13:26:11.2 (58ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:93 @ 09/18/26 13:26:11.2 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:93 @ 09/18/26 13:26:11.458 (257ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:111 @ 09/18/26 13:26:11.458 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:111 @ 09/18/26 13:30:27.114 (4m15.656s) < Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:79 @ 09/18/26 13:30:27.114 (4m15.971s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:30:27.114 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:30:27.121 (7ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:30:27.121 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:30:27.129 (8ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:140 @ 09/18/26 13:30:27.129 [SKIPPED] MPI runtime image is not available In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:142 @ 09/18/26 13:30:27.129 < Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:140 @ 09/18/26 13:30:27.129 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:30:27.129 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:30:27.14 (11ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:30:27.14 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:30:27.156 (16ms) > Enter [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:197 @ 09/18/26 13:30:27.156 STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:203 @ 09/18/26 13:30:27.156 END STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:203 @ 09/18/26 13:30:27.206 (50ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:208 @ 09/18/26 13:30:27.206 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:208 @ 09/18/26 13:30:27.466 (260ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:226 @ 09/18/26 13:30:27.466 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:226 @ 09/18/26 13:32:53.721 (2m26.256s) < Exit [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:197 @ 09/18/26 13:32:53.721 (2m26.565s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:32:53.721 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:32:53.726 (5ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:32:53.727 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:32:53.737 (10ms) > Enter [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:255 @ 09/18/26 13:32:53.737 STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:261 @ 09/18/26 13:32:53.737 END STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:261 @ 09/18/26 13:32:53.788 (51ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:266 @ 09/18/26 13:32:53.788 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:266 @ 09/18/26 13:32:54.045 (257ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:284 @ 09/18/26 13:32:54.045 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:284 @ 09/18/26 13:33:36.358 (42.313s) < Exit [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:255 @ 09/18/26 13:33:36.358 (42.621s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:36.358 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:33:36.365 (7ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:36.366 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:33:36.376 (10ms) > Enter [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:312 @ 09/18/26 13:33:36.376 STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:331 @ 09/18/26 13:33:36.376 END STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:331 @ 09/18/26 13:33:36.395 (19ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:335 @ 09/18/26 13:33:36.395 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:335 @ 09/18/26 13:33:36.658 (263ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:352 @ 09/18/26 13:33:36.658 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:352 @ 09/18/26 13:34:05.433 (28.775s) < Exit [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:312 @ 09/18/26 13:34:05.433 (29.057s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:05.433 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:05.44 (7ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:05.44 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:05.449 (9ms) > Enter [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:380 @ 09/18/26 13:34:05.449 STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:432 @ 09/18/26 13:34:05.449 END STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:432 @ 09/18/26 13:34:05.471 (22ms) STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:436 @ 09/18/26 13:34:05.471 END STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:436 @ 09/18/26 13:34:05.479 (8ms) < Exit [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:380 @ 09/18/26 13:34:05.479 (30ms) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:05.479 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:05.487 (8ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:05.487 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:05.499 (11ms) > Enter [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:450 @ 09/18/26 13:34:05.499 STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:504 @ 09/18/26 13:34:05.499 END STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:504 @ 09/18/26 13:34:05.522 (23ms) STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:509 @ 09/18/26 13:34:05.522 END STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:509 @ 09/18/26 13:34:05.527 (5ms) STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:526 @ 09/18/26 13:34:06.527 END STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:526 @ 09/18/26 13:34:06.551 (24ms) STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:545 @ 09/18/26 13:34:06.551 END STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:545 @ 09/18/26 13:34:06.555 (4ms) < Exit [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:450 @ 09/18/26 13:34:06.555 (1.056s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:06.555 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:06.562 (7ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:06.562 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:06.57 (8ms) > Enter [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:567 @ 09/18/26 13:34:06.57 STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:574 @ 09/18/26 13:34:06.57 END STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:574 @ 09/18/26 13:34:06.583 (13ms) STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:578 @ 09/18/26 13:34:06.583 END STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:578 @ 09/18/26 13:34:21.584 (15.001s) < Exit [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:567 @ 09/18/26 13:34:21.584 (15.014s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:21.584 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:21.589 (5ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:21.589 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:21.6 (11ms) > Enter [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:590 @ 09/18/26 13:34:21.6 STEP: Waiting for TrainJob to fail due to deadline - /workspace/source/src_code/test/e2e/e2e_test.go:605 @ 09/18/26 13:34:21.617 STEP: Ensuring the underlying JobSet is deleted - /workspace/source/src_code/test/e2e/e2e_test.go:612 @ 09/18/26 13:34:31.042 < Exit [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:590 @ 09/18/26 13:34:31.047 (9.447s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:31.047 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:31.059 (12ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:31.059 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:31.069 (10ms) > Enter [It] should inject runtime configuration which allows the runtime status endpoint to be called - /workspace/source/src_code/test/e2e/e2e_test.go:620 @ 09/18/26 13:34:31.069 [SKIPPED] TrainJobStatus feature gate is not enabled In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:622 @ 09/18/26 13:34:31.07 < Exit [It] should inject runtime configuration which allows the runtime status endpoint to be called - /workspace/source/src_code/test/e2e/e2e_test.go:620 @ 09/18/26 13:34:31.07 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:31.07 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:31.08 (10ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:31.08 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:31.096 (15ms) > Enter [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:846 @ 09/18/26 13:34:31.096 STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:738 @ 09/18/26 13:34:31.096 END STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:738 @ 09/18/26 13:34:31.135 (39ms) STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:751 @ 09/18/26 13:34:31.135 END STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:751 @ 09/18/26 13:34:31.149 (14ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:755 @ 09/18/26 13:34:31.149 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:755 @ 09/18/26 13:34:31.411 (261ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/18/26 13:34:31.411 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/18/26 13:34:31.428 (17ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:775 @ 09/18/26 13:34:31.428 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:775 @ 09/18/26 13:34:31.685 (258ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:788 @ 09/18/26 13:34:31.685 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:788 @ 09/18/26 13:34:31.708 (22ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:809 @ 09/18/26 13:34:31.708 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:809 @ 09/18/26 13:34:31.726 (18ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:820 @ 09/18/26 13:34:31.726 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:820 @ 09/18/26 13:34:31.984 (258ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:832 @ 09/18/26 13:34:31.984 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:832 @ 09/18/26 13:34:32.251 (267ms) < Exit [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:846 @ 09/18/26 13:34:32.251 (1.155s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:32.251 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:32.258 (7ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/18/26 13:34:32.258 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/18/26 13:34:32.263 (5ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:32.263 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 13:34:32.271 (8ms) > Enter [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:858 @ 09/18/26 13:34:32.271 STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:738 @ 09/18/26 13:34:32.271 END STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:738 @ 09/18/26 13:34:32.298 (27ms) STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:751 @ 09/18/26 13:34:32.298 END STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:751 @ 09/18/26 13:34:32.312 (14ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:755 @ 09/18/26 13:34:32.312 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:755 @ 09/18/26 13:34:32.569 (257ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/18/26 13:34:32.569 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:766 @ 09/18/26 13:34:32.587 (18ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:775 @ 09/18/26 13:34:32.587 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:775 @ 09/18/26 13:34:32.845 (258ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:788 @ 09/18/26 13:34:32.845 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:788 @ 09/18/26 13:34:32.862 (16ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:809 @ 09/18/26 13:34:32.862 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:809 @ 09/18/26 13:34:32.88 (18ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:820 @ 09/18/26 13:34:32.88 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:820 @ 09/18/26 13:34:33.138 (258ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:832 @ 09/18/26 13:34:33.138 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:832 @ 09/18/26 13:34:33.404 (266ms) < Exit [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:858 @ 09/18/26 13:34:33.404 (1.133s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:33.404 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 13:34:33.41 (6ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/18/26 13:34:33.41 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:742 @ 09/18/26 13:34:33.416 (6ms) > Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:45 @ 09/18/26 13:34:33.435 Created shared test namespace: rhai-e2e-progression-68r6s < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:45 @ 09/18/26 13:34:33.483 (49ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:34:33.483 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:34:33.509 (26ms) > Enter [It] should poll metrics and update trainerStatus annotation during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:150 @ 09/18/26 13:34:33.509 STEP: Creating TrainJob with progression tracking enabled - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:172 @ 09/18/26 13:34:33.509 STEP: Waiting for TrainJob pod to be running with IP - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:175 @ 09/18/26 13:34:33.538 STEP: Waiting for trainerStatus annotation to be populated - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:195 @ 09/18/26 13:35:31.712 STEP: Verifying trainerStatus is continuously updated during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:222 @ 09/18/26 13:35:33.721 STEP: Waiting for TrainJob to complete and verify final status - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:241 @ 09/18/26 13:35:33.725 < Exit [It] should poll metrics and update trainerStatus annotation during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:150 @ 09/18/26 13:35:55.786 (1m22.277s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:35:55.786 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:35:55.792 (6ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:35:55.792 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:35:55.804 (12ms) > Enter [It] should NOT create trainerStatus annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:274 @ 09/18/26 13:35:55.804 STEP: Creating TrainJob without progression tracking annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:289 @ 09/18/26 13:35:55.804 STEP: Verifying trainerStatus annotation is NOT created (checking over time) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:292 @ 09/18/26 13:35:55.818 < Exit [It] should NOT create trainerStatus annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:274 @ 09/18/26 13:36:30.834 (35.03s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:30.834 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:30.841 (7ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:30.842 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:30.855 (14ms) > Enter [It] should NOT enable progression tracking for non-'true' values - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:307 @ 09/18/26 13:36:30.856 STEP: Creating TrainJob with invalid annotation value - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:324 @ 09/18/26 13:36:30.856 STEP: Verifying progression tracking is NOT enabled (checking over time) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:327 @ 09/18/26 13:36:30.869 < Exit [It] should NOT enable progression tracking for non-'true' values - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:307 @ 09/18/26 13:36:45.875 (15.019s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:45.875 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:45.883 (9ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:45.883 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:45.899 (16ms) > Enter [It] should honor custom metrics port and poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:342 @ 09/18/26 13:36:45.899 STEP: Creating TrainJob with custom metrics configuration - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:360 @ 09/18/26 13:36:45.899 STEP: Verifying custom configuration is applied - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:363 @ 09/18/26 13:36:45.915 STEP: Verifying metrics are polled on the custom port - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:370 @ 09/18/26 13:36:45.919 < Exit [It] should honor custom metrics port and poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:342 @ 09/18/26 13:36:49.933 (4.034s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:49.933 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:49.939 (6ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:49.939 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:49.954 (15ms) > Enter [It] should handle minimum SDK-recommended poll interval (5s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:385 @ 09/18/26 13:36:49.954 STEP: Creating TrainJob with minimum poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:403 @ 09/18/26 13:36:49.954 STEP: Verifying minimum interval is respected - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:406 @ 09/18/26 13:36:49.968 < Exit [It] should handle minimum SDK-recommended poll interval (5s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:385 @ 09/18/26 13:36:49.972 (18ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:49.972 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:49.98 (8ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:49.98 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:49.996 (15ms) > Enter [It] should handle maximum SDK-recommended poll interval (300s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:412 @ 09/18/26 13:36:49.996 STEP: Creating TrainJob with maximum poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:430 @ 09/18/26 13:36:49.996 STEP: Verifying maximum interval is respected - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:433 @ 09/18/26 13:36:50.01 < Exit [It] should handle maximum SDK-recommended poll interval (300s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:412 @ 09/18/26 13:36:50.013 (18ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:50.013 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:50.019 (5ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:50.019 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:50.032 (13ms) > Enter [It] should use default interval when annotation is missing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:439 @ 09/18/26 13:36:50.032 STEP: Creating TrainJob without poll interval annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:457 @ 09/18/26 13:36:50.032 STEP: Verifying default interval (30s) is used - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:460 @ 09/18/26 13:36:50.048 < Exit [It] should use default interval when annotation is missing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:439 @ 09/18/26 13:36:50.053 (21ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:50.053 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:36:50.061 (8ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:50.061 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:36:50.077 (16ms) > Enter [BeforeEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:470 @ 09/18/26 13:36:50.077 < Exit [BeforeEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:470 @ 09/18/26 13:36:50.092 (15ms) > Enter [It] should capture final status even when job fails - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:487 @ 09/18/26 13:36:50.092 STEP: Creating TrainJob that will fail mid-training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:505 @ 09/18/26 13:36:50.092 STEP: Waiting for trainerStatus annotation to be populated during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:508 @ 09/18/26 13:36:50.106 STEP: Waiting for TrainJob to fail - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:516 @ 09/18/26 13:37:54.289 STEP: Verifying final status is captured with progress=100% even on failure - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:530 @ 09/18/26 13:38:04.322 < Exit [It] should capture final status even when job fails - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:487 @ 09/18/26 13:38:04.325 (1m14.233s) > Enter [AfterEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:481 @ 09/18/26 13:38:04.325 < Exit [AfterEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:481 @ 09/18/26 13:38:04.33 (5ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:38:04.33 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:38:04.336 (6ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:38:04.336 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 13:38:04.35 (14ms) > Enter [BeforeEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:550 @ 09/18/26 13:38:04.35 < Exit [BeforeEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:550 @ 09/18/26 13:38:04.361 (12ms) > Enter [It] should handle connection errors gracefully without crashing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:567 @ 09/18/26 13:38:04.361 STEP: Creating TrainJob without metrics endpoint - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:585 @ 09/18/26 13:38:04.361 STEP: Waiting for TrainJob pod to be running - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:588 @ 09/18/26 13:38:04.375 STEP: Check controller still reconciles the TrainJob by ensuring it gets marked complete - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:598 @ 09/18/26 13:38:06.461 < Exit [It] should handle connection errors gracefully without crashing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:567 @ 09/18/26 13:38:14.487 (10.125s) > Enter [AfterEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:561 @ 09/18/26 13:38:14.487 < Exit [AfterEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:561 @ 09/18/26 13:38:14.494 (7ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:38:14.494 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 13:38:14.499 (6ms) > Enter [AfterSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:71 @ 09/18/26 13:38:14.499 Cleaning up test namespace: rhai-e2e-progression-68r6s < Exit [AfterSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:71 @ 09/18/26 13:38:14.507 (7ms)