> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:42 @ 10/30/26 13:48:08.18 < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:42 @ 10/30/26 13:48:08.182 (2ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 10/30/26 13:48:08.182 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 10/30/26 13:48:08.315 (133ms) > Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:51 @ 10/30/26 13:48:08.315 STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:60 @ 10/30/26 13:48:08.315 END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:60 @ 10/30/26 13:48:08.39 (75ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:65 @ 10/30/26 13:48:08.39 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:65 @ 10/30/26 13:48:08.937 (547ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:83 @ 10/30/26 13:48:08.937 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:83 @ 10/30/26 13:51:22.03 (3m13.093s) < Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:51 @ 10/30/26 13:51:22.03 (3m13.715s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 10/30/26 13:51:22.03 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 10/30/26 13:51:22.047 (18ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 10/30/26 13:51:22.048 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 10/30/26 13:51:22.078 (30ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:112 @ 10/30/26 13:51:22.078 [SKIPPED] MPI runtime image is not available In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:114 @ 10/30/26 13:51:22.078 < Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:112 @ 10/30/26 13:51:22.078 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 10/30/26 13:51:22.078 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 10/30/26 13:51:22.094 (16ms) > Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:45 @ 10/30/26 13:51:22.114 Created shared test namespace: rhai-e2e-progression-kxmcm < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:45 @ 10/30/26 13:51:22.224 (110ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:51:22.224 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:51:22.311 (87ms) > Enter [It] should poll metrics and update trainerStatus annotation during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:94 @ 10/30/26 13:51:22.311 STEP: Creating TrainJob with progression tracking enabled - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:116 @ 10/30/26 13:51:22.311 STEP: Waiting for TrainJob pod to be running with IP - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:119 @ 10/30/26 13:51:22.376 STEP: Waiting for trainerStatus annotation to be populated - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:139 @ 10/30/26 13:53:09.131 STEP: Verifying trainerStatus is continuously updated during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:166 @ 10/30/26 13:53:11.157 STEP: Waiting for TrainJob to complete and verify final status - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:185 @ 10/30/26 13:53:11.17 < Exit [It] should poll metrics and update trainerStatus annotation during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:94 @ 10/30/26 13:53:35.354 (2m13.043s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:53:35.354 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:53:35.371 (17ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:53:35.371 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:53:35.459 (88ms) > Enter [It] should NOT create trainerStatus annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:218 @ 10/30/26 13:53:35.459 STEP: Creating TrainJob without progression tracking annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:233 @ 10/30/26 13:53:35.459 STEP: Verifying trainerStatus annotation is NOT created (checking over time) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:236 @ 10/30/26 13:53:35.487 < Exit [It] should NOT create trainerStatus annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:218 @ 10/30/26 13:54:10.502 (35.043s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:54:10.502 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:54:10.519 (17ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:54:10.519 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:54:10.565 (46ms) > Enter [It] should NOT enable progression tracking for non-'true' values - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:251 @ 10/30/26 13:54:10.565 STEP: Creating TrainJob with invalid annotation value - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:268 @ 10/30/26 13:54:10.565 STEP: Verifying progression tracking is NOT enabled (checking over time) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:271 @ 10/30/26 13:54:10.595 < Exit [It] should NOT enable progression tracking for non-'true' values - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:251 @ 10/30/26 13:54:25.599 (15.033s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:54:25.599 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:54:25.624 (25ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:54:25.624 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:54:25.676 (52ms) > Enter [It] should honor custom metrics port and poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:286 @ 10/30/26 13:54:25.676 STEP: Creating TrainJob with custom metrics configuration - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:304 @ 10/30/26 13:54:25.676 STEP: Verifying custom configuration is applied - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:307 @ 10/30/26 13:54:25.708 < Exit [It] should honor custom metrics port and poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:286 @ 10/30/26 13:54:25.721 (45ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:54:25.721 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:54:25.738 (17ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:54:25.738 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:54:25.784 (46ms) > Enter [It] should handle minimum SDK-recommended poll interval (5s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:315 @ 10/30/26 13:54:25.784 STEP: Creating TrainJob with minimum poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:333 @ 10/30/26 13:54:25.784 STEP: Verifying minimum interval is respected - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:336 @ 10/30/26 13:54:25.812 < Exit [It] should handle minimum SDK-recommended poll interval (5s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:315 @ 10/30/26 13:54:25.824 (40ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:54:25.824 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:54:25.839 (15ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:54:25.839 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:54:25.909 (70ms) > Enter [It] should handle maximum SDK-recommended poll interval (300s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:342 @ 10/30/26 13:54:25.909 STEP: Creating TrainJob with maximum poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:360 @ 10/30/26 13:54:25.909 STEP: Verifying maximum interval is respected - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:363 @ 10/30/26 13:54:25.954 < Exit [It] should handle maximum SDK-recommended poll interval (300s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:342 @ 10/30/26 13:54:25.97 (61ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:54:25.97 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:54:25.996 (26ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:54:25.996 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:54:26.044 (48ms) > Enter [It] should use default interval when annotation is missing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:369 @ 10/30/26 13:54:26.044 STEP: Creating TrainJob without poll interval annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:387 @ 10/30/26 13:54:26.044 STEP: Verifying default interval (30s) is used - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:390 @ 10/30/26 13:54:26.072 < Exit [It] should use default interval when annotation is missing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:369 @ 10/30/26 13:54:26.084 (40ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:54:26.084 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:54:26.1 (16ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:54:26.1 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:54:26.147 (48ms) > Enter [BeforeEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:400 @ 10/30/26 13:54:26.147 < Exit [BeforeEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:400 @ 10/30/26 13:54:26.195 (48ms) > Enter [It] should capture final status even when job fails - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:421 @ 10/30/26 13:54:26.195 STEP: Creating TrainJob that will fail mid-training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:439 @ 10/30/26 13:54:26.195 STEP: Waiting for trainerStatus annotation to be populated during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:442 @ 10/30/26 13:54:26.225 STEP: Waiting for TrainJob to fail - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:450 @ 10/30/26 13:56:06.998 STEP: Verifying final status is captured with progress=100% even on failure - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:464 @ 10/30/26 13:56:15.074 < Exit [It] should capture final status even when job fails - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:421 @ 10/30/26 13:56:15.091 (1m48.896s) > Enter [AfterEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:415 @ 10/30/26 13:56:15.091 < Exit [AfterEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:415 @ 10/30/26 13:56:15.11 (19ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:56:15.11 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:56:15.127 (16ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:56:15.127 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 10/30/26 13:56:15.21 (83ms) > Enter [BeforeEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:484 @ 10/30/26 13:56:15.21 < Exit [BeforeEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:484 @ 10/30/26 13:56:15.256 (47ms) > Enter [It] should handle connection errors gracefully without crashing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:505 @ 10/30/26 13:56:15.256 STEP: Creating TrainJob without metrics endpoint - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:523 @ 10/30/26 13:56:15.256 STEP: Waiting for TrainJob pod to be running - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:526 @ 10/30/26 13:56:15.285 STEP: Check controller still reconciles the TrainJob by ensuring it gets marked complete - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:536 @ 10/30/26 13:56:17.395 < Exit [It] should handle connection errors gracefully without crashing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:505 @ 10/30/26 13:56:25.472 (10.216s) > Enter [AfterEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:499 @ 10/30/26 13:56:25.472 < Exit [AfterEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:499 @ 10/30/26 13:56:25.491 (18ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:56:25.491 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 10/30/26 13:56:25.507 (16ms) > Enter [AfterSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:71 @ 10/30/26 13:56:25.508 Cleaning up test namespace: rhai-e2e-progression-kxmcm < Exit [AfterSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:71 @ 10/30/26 13:56:25.524 (17ms)