> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:42 @ 11/02/26 21:52:37.979 < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:42 @ 11/02/26 21:52:37.98 (1ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 11/02/26 21:52:37.98 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 11/02/26 21:52:38.127 (147ms) > Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:51 @ 11/02/26 21:52:38.127 STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:60 @ 11/02/26 21:52:38.127 END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:60 @ 11/02/26 21:52:38.329 (202ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:65 @ 11/02/26 21:52:38.329 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:65 @ 11/02/26 21:52:39.674 (1.345s) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:83 @ 11/02/26 21:52:39.674 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:83 @ 11/02/26 22:01:33.632 (8m53.958s) < Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:51 @ 11/02/26 22:01:33.632 (8m55.505s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 11/02/26 22:01:33.632 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 11/02/26 22:01:33.649 (17ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 11/02/26 22:01:33.649 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 11/02/26 22:01:33.684 (34ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:112 @ 11/02/26 22:01:33.684 [SKIPPED] MPI runtime image is not available In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:114 @ 11/02/26 22:01:33.684 < Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:112 @ 11/02/26 22:01:33.684 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 11/02/26 22:01:33.684 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 11/02/26 22:01:33.706 (22ms) > Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:45 @ 11/02/26 22:01:33.727 Created shared test namespace: rhai-e2e-progression-qgx6z < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:45 @ 11/02/26 22:01:33.881 (154ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:01:33.881 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:01:34.086 (205ms) > Enter [It] should poll metrics and update trainerStatus annotation during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:94 @ 11/02/26 22:01:34.086 STEP: Creating TrainJob with progression tracking enabled - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:116 @ 11/02/26 22:01:34.086 STEP: Waiting for TrainJob pod to be running with IP - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:119 @ 11/02/26 22:01:34.3 STEP: Waiting for trainerStatus annotation to be populated - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:139 @ 11/02/26 22:03:15.272 STEP: Verifying trainerStatus is continuously updated during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:166 @ 11/02/26 22:03:15.289 STEP: Waiting for TrainJob to complete and verify final status - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:185 @ 11/02/26 22:03:15.307 < Exit [It] should poll metrics and update trainerStatus annotation during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:94 @ 11/02/26 22:03:39.55 (2m5.464s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:03:39.55 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:03:39.572 (22ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:03:39.573 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:03:39.776 (203ms) > Enter [It] should NOT create trainerStatus annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:218 @ 11/02/26 22:03:39.776 STEP: Creating TrainJob without progression tracking annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:233 @ 11/02/26 22:03:39.776 STEP: Verifying trainerStatus annotation is NOT created (checking over time) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:236 @ 11/02/26 22:03:39.862 < Exit [It] should NOT create trainerStatus annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:218 @ 11/02/26 22:04:14.863 (35.087s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:04:14.863 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:04:14.886 (23ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:04:14.886 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:04:14.975 (90ms) > Enter [It] should NOT enable progression tracking for non-'true' values - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:251 @ 11/02/26 22:04:14.975 STEP: Creating TrainJob with invalid annotation value - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:268 @ 11/02/26 22:04:14.976 STEP: Verifying progression tracking is NOT enabled (checking over time) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:271 @ 11/02/26 22:04:15.044 < Exit [It] should NOT enable progression tracking for non-'true' values - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:251 @ 11/02/26 22:04:30.046 (15.071s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:04:30.047 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:04:30.069 (23ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:04:30.069 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:04:30.16 (90ms) > Enter [It] should honor custom metrics port and poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:286 @ 11/02/26 22:04:30.16 STEP: Creating TrainJob with custom metrics configuration - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:304 @ 11/02/26 22:04:30.16 STEP: Verifying custom configuration is applied - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:307 @ 11/02/26 22:04:30.23 < Exit [It] should honor custom metrics port and poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:286 @ 11/02/26 22:04:30.247 (87ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:04:30.247 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:04:30.266 (20ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:04:30.267 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:04:30.356 (89ms) > Enter [It] should handle minimum SDK-recommended poll interval (5s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:315 @ 11/02/26 22:04:30.356 STEP: Creating TrainJob with minimum poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:333 @ 11/02/26 22:04:30.356 STEP: Verifying minimum interval is respected - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:336 @ 11/02/26 22:04:30.426 < Exit [It] should handle minimum SDK-recommended poll interval (5s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:315 @ 11/02/26 22:04:30.443 (87ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:04:30.443 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:04:30.463 (20ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:04:30.463 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:04:30.552 (90ms) > Enter [It] should handle maximum SDK-recommended poll interval (300s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:342 @ 11/02/26 22:04:30.552 STEP: Creating TrainJob with maximum poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:360 @ 11/02/26 22:04:30.552 STEP: Verifying maximum interval is respected - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:363 @ 11/02/26 22:04:30.623 < Exit [It] should handle maximum SDK-recommended poll interval (300s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:342 @ 11/02/26 22:04:30.64 (88ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:04:30.64 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:04:30.662 (22ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:04:30.662 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:04:30.751 (89ms) > Enter [It] should use default interval when annotation is missing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:369 @ 11/02/26 22:04:30.751 STEP: Creating TrainJob without poll interval annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:387 @ 11/02/26 22:04:30.751 STEP: Verifying default interval (30s) is used - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:390 @ 11/02/26 22:04:30.819 < Exit [It] should use default interval when annotation is missing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:369 @ 11/02/26 22:04:30.836 (86ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:04:30.836 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:04:30.856 (20ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:04:30.856 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:04:30.946 (89ms) > Enter [BeforeEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:400 @ 11/02/26 22:04:30.946 < Exit [BeforeEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:400 @ 11/02/26 22:04:31.035 (89ms) > Enter [It] should capture final status even when job fails - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:421 @ 11/02/26 22:04:31.035 STEP: Creating TrainJob that will fail mid-training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:439 @ 11/02/26 22:04:31.035 STEP: Waiting for trainerStatus annotation to be populated during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:442 @ 11/02/26 22:04:31.103 STEP: Waiting for TrainJob to fail - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:450 @ 11/02/26 22:06:12.056 STEP: Verifying final status is captured with progress=100% even on failure - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:464 @ 11/02/26 22:06:20.148 < Exit [It] should capture final status even when job fails - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:421 @ 11/02/26 22:06:20.166 (1m49.131s) > Enter [AfterEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:415 @ 11/02/26 22:06:20.166 < Exit [AfterEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:415 @ 11/02/26 22:06:20.188 (22ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:06:20.188 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:06:20.208 (20ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:06:20.208 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 22:06:20.408 (200ms) > Enter [BeforeEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:484 @ 11/02/26 22:06:20.408 < Exit [BeforeEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:484 @ 11/02/26 22:06:20.494 (86ms) > Enter [It] should handle connection errors gracefully without crashing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:505 @ 11/02/26 22:06:20.494 STEP: Creating TrainJob without metrics endpoint - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:523 @ 11/02/26 22:06:20.494 STEP: Waiting for TrainJob pod to be running - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:526 @ 11/02/26 22:06:20.57 STEP: Check controller still reconciles the TrainJob by ensuring it gets marked complete - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:536 @ 11/02/26 22:06:24.871 < Exit [It] should handle connection errors gracefully without crashing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:505 @ 11/02/26 22:06:32.964 (12.469s) > Enter [AfterEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:499 @ 11/02/26 22:06:32.964 < Exit [AfterEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:499 @ 11/02/26 22:06:32.984 (20ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:06:32.984 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 22:06:33.004 (20ms) > Enter [AfterSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:71 @ 11/02/26 22:06:33.004 Cleaning up test namespace: rhai-e2e-progression-qgx6z < Exit [AfterSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:71 @ 11/02/26 22:06:33.023 (19ms) === RUN TestKueueSchedulingReachesTrainJobPods kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK kueue_operator.go:227: Kueue CR 'cluster' exists kueue_operator.go:229: Waiting for Kueue CR to be ready... kueue_operator.go:236: Kueue CR is ready kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR... kueue_operator.go:243: TrainJob framework is present in Kueue CR trainer_kueue_integration_test.go:251: Created Kueue-managed namespace: test-ns-hb254 trainer_kueue_integration_test.go:259: Created Kueue ResourceFlavor rf-gv79f successfully trainer_kueue_integration_test.go:274: Created Kueue ClusterQueue cq-52pvb successfully trainer_kueue_integration_test.go:298: Created Kueue LocalQueue test-ns-hb254/lq-722jg successfully trainer_kueue_integration_test.go:332: Created TrainJob: test-runtimepatches-trainjob-pdk5q trainer_kueue_integration_test.go:340: Workload admitted trainer_kueue_integration_test.go:351: replicated job "node" should carry the ResourceFlavor's node label Expected <map[string]string | len:0>: nil to have {key: value} <map[interface {}]interface {} | len:1>: { <string>"kueue-test-flavor": <string>"on", } test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueSchedulingReachesTrainJobPods3132809584 kueue_operator.go:100: Collecting Kueue diagnostics... kueue_operator.go:100: Stored Kueue CR state kueue_operator.go:100: DataScienceCluster is not installed; skipping DSC diagnostics kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-5699f6b9b6-h2ckm/manager kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-5699f6b9b6-h4fbj/manager --- FAIL: TestKueueSchedulingReachesTrainJobPods (2.16s)