> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 09/18/26 08:59:33.222 < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:43 @ 09/18/26 08:59:33.223 (1ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 08:59:33.223 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 08:59:33.272 (49ms) > Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:79 @ 09/18/26 08:59:33.272 STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:85 @ 09/18/26 08:59:33.272 END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:85 @ 09/18/26 08:59:33.324 (52ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:90 @ 09/18/26 08:59:33.324 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:90 @ 09/18/26 08:59:33.585 (260ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:108 @ 09/18/26 08:59:33.585 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:108 @ 09/18/26 09:00:34.365 (1m0.781s) < Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:79 @ 09/18/26 09:00:34.365 (1m1.093s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:00:34.365 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:00:34.37 (5ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:00:34.37 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:00:34.38 (9ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:137 @ 09/18/26 09:00:34.38 [SKIPPED] MPI runtime image is not available In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:139 @ 09/18/26 09:00:34.38 < Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:137 @ 09/18/26 09:00:34.38 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:00:34.38 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:00:34.393 (14ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:00:34.394 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:00:34.408 (15ms) > Enter [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:194 @ 09/18/26 09:00:34.408 STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:200 @ 09/18/26 09:00:34.408 END STEP: Create a TrainJob with jax-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:200 @ 09/18/26 09:00:34.425 (17ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:205 @ 09/18/26 09:00:34.425 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:205 @ 09/18/26 09:00:34.682 (257ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:223 @ 09/18/26 09:00:34.682 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:223 @ 09/18/26 09:02:52.942 (2m18.261s) < Exit [It] should create TrainJob with JAX runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:194 @ 09/18/26 09:02:52.942 (2m18.534s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:02:52.942 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:02:52.949 (7ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:02:52.95 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:02:52.957 (8ms) > Enter [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:252 @ 09/18/26 09:02:52.957 STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:258 @ 09/18/26 09:02:52.957 END STEP: Create a TrainJob with xgboost-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:258 @ 09/18/26 09:02:52.997 (40ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:263 @ 09/18/26 09:02:52.997 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:263 @ 09/18/26 09:02:53.254 (257ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:281 @ 09/18/26 09:02:53.254 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:281 @ 09/18/26 09:03:36.227 (42.973s) < Exit [It] should create TrainJob with XGBoost runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:252 @ 09/18/26 09:03:36.227 (43.27s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:03:36.227 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:03:36.232 (5ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:03:36.233 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:03:36.24 (8ms) > Enter [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:309 @ 09/18/26 09:03:36.24 STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:328 @ 09/18/26 09:03:36.24 END STEP: Create a TrainJob with flux-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:328 @ 09/18/26 09:03:36.257 (17ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:332 @ 09/18/26 09:03:36.257 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:332 @ 09/18/26 09:03:36.515 (258ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:349 @ 09/18/26 09:03:36.515 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:349 @ 09/18/26 09:04:04.235 (27.72s) < Exit [It] should create TrainJob with Flux runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:309 @ 09/18/26 09:04:04.235 (27.995s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:04.235 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:04.243 (8ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:04.244 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:04.254 (10ms) > Enter [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:377 @ 09/18/26 09:04:04.254 STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:429 @ 09/18/26 09:04:04.254 END STEP: Create a TrainJob with RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:429 @ 09/18/26 09:04:04.284 (30ms) STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:433 @ 09/18/26 09:04:04.284 END STEP: Verify user-provided Time is preserved for test.io/manager-one and Time is set for kueue.k8s.io/manager - /workspace/source/src_code/test/e2e/e2e_test.go:433 @ 09/18/26 09:04:04.297 (13ms) < Exit [It] should preserve user-provided manager fields - /workspace/source/src_code/test/e2e/e2e_test.go:377 @ 09/18/26 09:04:04.297 (43ms) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:04.297 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:04.308 (12ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:04.309 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:04.331 (22ms) > Enter [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:447 @ 09/18/26 09:04:04.331 STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:501 @ 09/18/26 09:04:04.331 END STEP: Create a suspended TrainJob with two RuntimePatches - /workspace/source/src_code/test/e2e/e2e_test.go:501 @ 09/18/26 09:04:04.351 (20ms) STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:506 @ 09/18/26 09:04:04.351 END STEP: Record the original Time values and verify user-provided Time is preserved - /workspace/source/src_code/test/e2e/e2e_test.go:506 @ 09/18/26 09:04:04.359 (8ms) STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:523 @ 09/18/26 09:04:05.359 END STEP: Update the second RuntimePatch and add two new patches - /workspace/source/src_code/test/e2e/e2e_test.go:523 @ 09/18/26 09:04:05.386 (27ms) STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:542 @ 09/18/26 09:04:05.386 END STEP: Verify Time behaviour for all patches after update - /workspace/source/src_code/test/e2e/e2e_test.go:542 @ 09/18/26 09:04:05.39 (4ms) < Exit [It] should update Time only for the changed patch on update - /workspace/source/src_code/test/e2e/e2e_test.go:447 @ 09/18/26 09:04:05.39 (1.059s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:05.39 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:05.399 (9ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:05.399 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:05.409 (10ms) > Enter [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:564 @ 09/18/26 09:04:05.409 STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:571 @ 09/18/26 09:04:05.409 END STEP: Create a TrainJob managed by MultiKueue - /workspace/source/src_code/test/e2e/e2e_test.go:571 @ 09/18/26 09:04:05.426 (17ms) STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:575 @ 09/18/26 09:04:05.426 END STEP: Ensuring the built-in controller neither creates a JobSet nor sets any status - /workspace/source/src_code/test/e2e/e2e_test.go:575 @ 09/18/26 09:04:20.426 (15.001s) < Exit [It] should not be reconciled by the built-in controller - /workspace/source/src_code/test/e2e/e2e_test.go:564 @ 09/18/26 09:04:20.426 (15.018s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:20.426 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:20.434 (7ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:20.434 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:20.442 (8ms) > Enter [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:587 @ 09/18/26 09:04:20.442 STEP: Waiting for TrainJob to fail due to deadline - /workspace/source/src_code/test/e2e/e2e_test.go:602 @ 09/18/26 09:04:20.46 STEP: Ensuring the underlying JobSet is deleted - /workspace/source/src_code/test/e2e/e2e_test.go:609 @ 09/18/26 09:04:30.148 < Exit [It] should fail the TrainJob with DeadlineExceeded when active timeout expires - /workspace/source/src_code/test/e2e/e2e_test.go:587 @ 09/18/26 09:04:30.151 (9.709s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:30.151 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:30.16 (8ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:30.16 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:30.178 (18ms) > Enter [It] should inject runtime configuration which allows the runtime status endpoint to be called - /workspace/source/src_code/test/e2e/e2e_test.go:617 @ 09/18/26 09:04:30.178 [SKIPPED] TrainJobStatus feature gate is not enabled In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:619 @ 09/18/26 09:04:30.178 < Exit [It] should inject runtime configuration which allows the runtime status endpoint to be called - /workspace/source/src_code/test/e2e/e2e_test.go:617 @ 09/18/26 09:04:30.179 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:30.179 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:30.214 (35ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:30.214 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:30.233 (20ms) > Enter [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:843 @ 09/18/26 09:04:30.233 STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:735 @ 09/18/26 09:04:30.233 END STEP: Creating a TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:735 @ 09/18/26 09:04:30.289 (55ms) STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:748 @ 09/18/26 09:04:30.289 END STEP: Creating a TrainJob that references the TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:748 @ 09/18/26 09:04:30.316 (27ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:752 @ 09/18/26 09:04:30.316 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:752 @ 09/18/26 09:04:30.577 (261ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:763 @ 09/18/26 09:04:30.577 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:763 @ 09/18/26 09:04:30.603 (25ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:772 @ 09/18/26 09:04:30.603 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:772 @ 09/18/26 09:04:30.86 (257ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:785 @ 09/18/26 09:04:30.86 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:785 @ 09/18/26 09:04:30.876 (15ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:806 @ 09/18/26 09:04:30.876 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:806 @ 09/18/26 09:04:30.896 (21ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:817 @ 09/18/26 09:04:30.896 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:817 @ 09/18/26 09:04:31.154 (258ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:829 @ 09/18/26 09:04:31.154 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:829 @ 09/18/26 09:04:31.419 (265ms) < Exit [It] TrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:843 @ 09/18/26 09:04:31.419 (1.186s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:31.419 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:31.425 (6ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:739 @ 09/18/26 09:04:31.425 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:739 @ 09/18/26 09:04:31.432 (7ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:31.432 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:56 @ 09/18/26 09:04:31.443 (11ms) > Enter [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:855 @ 09/18/26 09:04:31.443 STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:735 @ 09/18/26 09:04:31.443 END STEP: Creating a ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:735 @ 09/18/26 09:04:31.471 (27ms) STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:748 @ 09/18/26 09:04:31.471 END STEP: Creating a TrainJob that references the ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:748 @ 09/18/26 09:04:31.486 (15ms) STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:752 @ 09/18/26 09:04:31.486 END STEP: Waiting for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:752 @ 09/18/26 09:04:31.75 (264ms) STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:763 @ 09/18/26 09:04:31.75 END STEP: Pausing the TrainJob - /workspace/source/src_code/test/e2e/e2e_test.go:763 @ 09/18/26 09:04:31.768 (18ms) STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:772 @ 09/18/26 09:04:31.768 END STEP: Waiting for TrainJob to be suspended - /workspace/source/src_code/test/e2e/e2e_test.go:772 @ 09/18/26 09:04:32.026 (258ms) STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:785 @ 09/18/26 09:04:32.026 END STEP: Updating the runtime with a new container image - /workspace/source/src_code/test/e2e/e2e_test.go:785 @ 09/18/26 09:04:32.039 (13ms) STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:806 @ 09/18/26 09:04:32.039 END STEP: Restarting the TrainJob by unpausing it - /workspace/source/src_code/test/e2e/e2e_test.go:806 @ 09/18/26 09:04:32.055 (16ms) STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:817 @ 09/18/26 09:04:32.055 END STEP: Verifying the TrainJob jobs become active after resuming - /workspace/source/src_code/test/e2e/e2e_test.go:817 @ 09/18/26 09:04:32.312 (258ms) STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:829 @ 09/18/26 09:04:32.312 END STEP: Verifying the Runtime snapshot is garbage collected - /workspace/source/src_code/test/e2e/e2e_test.go:829 @ 09/18/26 09:04:32.6 (287ms) < Exit [It] ClusterTrainingRuntime - /workspace/source/src_code/test/e2e/e2e_test.go:855 @ 09/18/26 09:04:32.6 (1.156s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:32.6 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:71 @ 09/18/26 09:04:32.605 (5ms) > Enter [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:739 @ 09/18/26 09:04:32.605 < Exit [DeferCleanup (Each)] a suspended TrainJob should use the original runtime configuration and not pick up the new configuration - /workspace/source/src_code/test/e2e/e2e_test.go:739 @ 09/18/26 09:04:32.61 (6ms) > Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:45 @ 09/18/26 09:04:32.632 Created shared test namespace: rhai-e2e-progression-t2hcm < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:45 @ 09/18/26 09:04:32.68 (48ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:04:32.68 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:04:32.702 (22ms) > Enter [It] should poll metrics and update trainerStatus annotation during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:150 @ 09/18/26 09:04:32.702 STEP: Creating TrainJob with progression tracking enabled - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:172 @ 09/18/26 09:04:32.702 STEP: Waiting for TrainJob pod to be running with IP - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:175 @ 09/18/26 09:04:32.725 STEP: Waiting for trainerStatus annotation to be populated - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:195 @ 09/18/26 09:05:32.925 STEP: Verifying trainerStatus is continuously updated during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:222 @ 09/18/26 09:05:32.929 STEP: Waiting for TrainJob to complete and verify final status - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:241 @ 09/18/26 09:05:32.932 < Exit [It] should poll metrics and update trainerStatus annotation during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:150 @ 09/18/26 09:05:57.002 (1m24.299s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:05:57.002 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:05:57.007 (6ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:05:57.008 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:05:57.021 (13ms) > Enter [It] should NOT create trainerStatus annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:274 @ 09/18/26 09:05:57.021 STEP: Creating TrainJob without progression tracking annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:289 @ 09/18/26 09:05:57.021 STEP: Verifying trainerStatus annotation is NOT created (checking over time) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:292 @ 09/18/26 09:05:57.036 < Exit [It] should NOT create trainerStatus annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:274 @ 09/18/26 09:06:32.052 (35.031s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:06:32.052 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:06:32.058 (7ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:06:32.059 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:06:32.074 (15ms) > Enter [It] should NOT enable progression tracking for non-'true' values - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:307 @ 09/18/26 09:06:32.074 STEP: Creating TrainJob with invalid annotation value - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:324 @ 09/18/26 09:06:32.074 STEP: Verifying progression tracking is NOT enabled (checking over time) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:327 @ 09/18/26 09:06:32.09 < Exit [It] should NOT enable progression tracking for non-'true' values - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:307 @ 09/18/26 09:06:47.092 (15.018s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:06:47.092 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:06:47.098 (6ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:06:47.098 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:06:47.111 (13ms) > Enter [It] should honor custom metrics port and poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:342 @ 09/18/26 09:06:47.111 STEP: Creating TrainJob with custom metrics configuration - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:360 @ 09/18/26 09:06:47.111 STEP: Verifying custom configuration is applied - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:363 @ 09/18/26 09:06:47.127 STEP: Verifying metrics are polled on the custom port - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:370 @ 09/18/26 09:06:47.131 < Exit [It] should honor custom metrics port and poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:342 @ 09/18/26 09:06:49.141 (2.03s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:06:49.141 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:06:49.146 (5ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:06:49.146 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:06:49.16 (13ms) > Enter [It] should handle minimum SDK-recommended poll interval (5s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:385 @ 09/18/26 09:06:49.16 STEP: Creating TrainJob with minimum poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:403 @ 09/18/26 09:06:49.16 STEP: Verifying minimum interval is respected - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:406 @ 09/18/26 09:06:49.173 < Exit [It] should handle minimum SDK-recommended poll interval (5s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:385 @ 09/18/26 09:06:49.176 (16ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:06:49.176 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:06:49.184 (8ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:06:49.184 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:06:49.194 (10ms) > Enter [It] should handle maximum SDK-recommended poll interval (300s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:412 @ 09/18/26 09:06:49.194 STEP: Creating TrainJob with maximum poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:430 @ 09/18/26 09:06:49.194 STEP: Verifying maximum interval is respected - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:433 @ 09/18/26 09:06:49.208 < Exit [It] should handle maximum SDK-recommended poll interval (300s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:412 @ 09/18/26 09:06:49.214 (19ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:06:49.214 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:06:49.219 (5ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:06:49.219 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:06:49.23 (11ms) > Enter [It] should use default interval when annotation is missing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:439 @ 09/18/26 09:06:49.23 STEP: Creating TrainJob without poll interval annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:457 @ 09/18/26 09:06:49.23 STEP: Verifying default interval (30s) is used - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:460 @ 09/18/26 09:06:49.245 < Exit [It] should use default interval when annotation is missing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:439 @ 09/18/26 09:06:49.248 (18ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:06:49.248 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:06:49.259 (11ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:06:49.259 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:06:49.275 (16ms) > Enter [BeforeEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:470 @ 09/18/26 09:06:49.275 < Exit [BeforeEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:470 @ 09/18/26 09:06:49.288 (13ms) > Enter [It] should capture final status even when job fails - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:487 @ 09/18/26 09:06:49.288 STEP: Creating TrainJob that will fail mid-training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:505 @ 09/18/26 09:06:49.288 STEP: Waiting for trainerStatus annotation to be populated during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:508 @ 09/18/26 09:06:49.301 STEP: Waiting for TrainJob to fail - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:516 @ 09/18/26 09:07:53.462 STEP: Verifying final status is captured with progress=100% even on failure - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:530 @ 09/18/26 09:08:05.495 < Exit [It] should capture final status even when job fails - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:487 @ 09/18/26 09:08:05.498 (1m16.21s) > Enter [AfterEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:481 @ 09/18/26 09:08:05.498 < Exit [AfterEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:481 @ 09/18/26 09:08:05.504 (5ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:08:05.504 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:08:05.509 (5ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:08:05.509 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:130 @ 09/18/26 09:08:05.523 (14ms) > Enter [BeforeEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:550 @ 09/18/26 09:08:05.523 < Exit [BeforeEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:550 @ 09/18/26 09:08:05.538 (15ms) > Enter [It] should handle connection errors gracefully without crashing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:567 @ 09/18/26 09:08:05.538 STEP: Creating TrainJob without metrics endpoint - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:585 @ 09/18/26 09:08:05.538 STEP: Waiting for TrainJob pod to be running - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:588 @ 09/18/26 09:08:05.554 STEP: Check controller still reconciles the TrainJob by ensuring it gets marked complete - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:598 @ 09/18/26 09:08:07.667 < Exit [It] should handle connection errors gracefully without crashing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:567 @ 09/18/26 09:08:15.689 (10.151s) > Enter [AfterEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:561 @ 09/18/26 09:08:15.689 < Exit [AfterEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:561 @ 09/18/26 09:08:15.694 (6ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:08:15.694 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:142 @ 09/18/26 09:08:15.701 (7ms) > Enter [AfterSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:71 @ 09/18/26 09:08:15.702 Cleaning up test namespace: rhai-e2e-progression-t2hcm < Exit [AfterSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:71 @ 09/18/26 09:08:15.707 (5ms)