> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:42 @ 11/02/26 21:45:06.36 < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:42 @ 11/02/26 21:45:06.361 (1ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 11/02/26 21:45:06.361 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 11/02/26 21:45:06.496 (135ms) > Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:51 @ 11/02/26 21:45:06.496 STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:60 @ 11/02/26 21:45:06.496 END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:60 @ 11/02/26 21:45:06.682 (186ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:65 @ 11/02/26 21:45:06.682 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:65 @ 11/02/26 21:45:07.752 (1.071s) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:83 @ 11/02/26 21:45:07.752 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:83 @ 11/02/26 21:53:33.358 (8m25.606s) < Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:51 @ 11/02/26 21:53:33.358 (8m26.862s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 11/02/26 21:53:33.358 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 11/02/26 21:53:33.373 (15ms) > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 11/02/26 21:53:33.373 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 11/02/26 21:53:33.404 (31ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:112 @ 11/02/26 21:53:33.404 [SKIPPED] MPI runtime image is not available In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:114 @ 11/02/26 21:53:33.405 < Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:112 @ 11/02/26 21:53:33.405 (0s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 11/02/26 21:53:33.405 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 11/02/26 21:53:33.427 (22ms) > Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:45 @ 11/02/26 21:53:33.446 Created shared test namespace: rhai-e2e-progression-sfndj < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:45 @ 11/02/26 21:53:33.576 (129ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:53:33.576 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:53:33.775 (199ms) > Enter [It] should poll metrics and update trainerStatus annotation during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:94 @ 11/02/26 21:53:33.775 STEP: Creating TrainJob with progression tracking enabled - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:116 @ 11/02/26 21:53:33.775 STEP: Waiting for TrainJob pod to be running with IP - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:119 @ 11/02/26 21:53:33.956 STEP: Waiting for trainerStatus annotation to be populated - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:139 @ 11/02/26 21:55:12.724 STEP: Verifying trainerStatus is continuously updated during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:166 @ 11/02/26 21:55:16.769 STEP: Waiting for TrainJob to complete and verify final status - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:185 @ 11/02/26 21:55:16.785 < Exit [It] should poll metrics and update trainerStatus annotation during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:94 @ 11/02/26 21:55:38.966 (2m5.191s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:55:38.966 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:55:38.985 (18ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:55:38.985 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:55:39.179 (194ms) > Enter [It] should NOT create trainerStatus annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:218 @ 11/02/26 21:55:39.179 STEP: Creating TrainJob without progression tracking annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:233 @ 11/02/26 21:55:39.179 STEP: Verifying trainerStatus annotation is NOT created (checking over time) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:236 @ 11/02/26 21:55:39.246 < Exit [It] should NOT create trainerStatus annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:218 @ 11/02/26 21:56:14.248 (35.07s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:56:14.249 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:56:14.267 (18ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:56:14.267 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:56:14.351 (84ms) > Enter [It] should NOT enable progression tracking for non-'true' values - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:251 @ 11/02/26 21:56:14.351 STEP: Creating TrainJob with invalid annotation value - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:268 @ 11/02/26 21:56:14.351 STEP: Verifying progression tracking is NOT enabled (checking over time) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:271 @ 11/02/26 21:56:14.419 < Exit [It] should NOT enable progression tracking for non-'true' values - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:251 @ 11/02/26 21:56:29.425 (15.073s) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:56:29.425 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:56:29.444 (19ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:56:29.444 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:56:29.529 (85ms) > Enter [It] should honor custom metrics port and poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:286 @ 11/02/26 21:56:29.529 STEP: Creating TrainJob with custom metrics configuration - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:304 @ 11/02/26 21:56:29.529 STEP: Verifying custom configuration is applied - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:307 @ 11/02/26 21:56:29.597 < Exit [It] should honor custom metrics port and poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:286 @ 11/02/26 21:56:29.612 (83ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:56:29.612 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:56:29.628 (16ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:56:29.628 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:56:29.715 (87ms) > Enter [It] should handle minimum SDK-recommended poll interval (5s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:315 @ 11/02/26 21:56:29.715 STEP: Creating TrainJob with minimum poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:333 @ 11/02/26 21:56:29.715 STEP: Verifying minimum interval is respected - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:336 @ 11/02/26 21:56:29.783 < Exit [It] should handle minimum SDK-recommended poll interval (5s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:315 @ 11/02/26 21:56:29.796 (81ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:56:29.796 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:56:29.812 (16ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:56:29.812 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:56:29.897 (84ms) > Enter [It] should handle maximum SDK-recommended poll interval (300s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:342 @ 11/02/26 21:56:29.897 STEP: Creating TrainJob with maximum poll interval - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:360 @ 11/02/26 21:56:29.897 STEP: Verifying maximum interval is respected - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:363 @ 11/02/26 21:56:29.966 < Exit [It] should handle maximum SDK-recommended poll interval (300s) - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:342 @ 11/02/26 21:56:29.98 (83ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:56:29.98 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:56:29.998 (19ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:56:29.998 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:56:30.083 (84ms) > Enter [It] should use default interval when annotation is missing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:369 @ 11/02/26 21:56:30.083 STEP: Creating TrainJob without poll interval annotation - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:387 @ 11/02/26 21:56:30.083 STEP: Verifying default interval (30s) is used - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:390 @ 11/02/26 21:56:30.15 < Exit [It] should use default interval when annotation is missing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:369 @ 11/02/26 21:56:30.164 (81ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:56:30.164 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:56:30.179 (15ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:56:30.179 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:56:30.264 (85ms) > Enter [BeforeEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:400 @ 11/02/26 21:56:30.264 < Exit [BeforeEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:400 @ 11/02/26 21:56:30.348 (84ms) > Enter [It] should capture final status even when job fails - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:421 @ 11/02/26 21:56:30.348 STEP: Creating TrainJob that will fail mid-training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:439 @ 11/02/26 21:56:30.348 STEP: Waiting for trainerStatus annotation to be populated during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:442 @ 11/02/26 21:56:30.415 STEP: Waiting for TrainJob to fail - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:450 @ 11/02/26 21:58:13.198 STEP: Verifying final status is captured with progress=100% even on failure - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:464 @ 11/02/26 21:58:21.274 < Exit [It] should capture final status even when job fails - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:421 @ 11/02/26 21:58:21.288 (1m50.94s) > Enter [AfterEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:415 @ 11/02/26 21:58:21.288 < Exit [AfterEach] When TrainJob fails during training - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:415 @ 11/02/26 21:58:21.306 (18ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:58:21.306 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:58:21.321 (16ms) > Enter [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:58:21.322 < Exit [BeforeEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:69 @ 11/02/26 21:58:21.517 (195ms) > Enter [BeforeEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:484 @ 11/02/26 21:58:21.517 < Exit [BeforeEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:484 @ 11/02/26 21:58:21.6 (83ms) > Enter [It] should handle connection errors gracefully without crashing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:505 @ 11/02/26 21:58:21.6 STEP: Creating TrainJob without metrics endpoint - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:523 @ 11/02/26 21:58:21.6 STEP: Waiting for TrainJob pod to be running - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:526 @ 11/02/26 21:58:21.668 STEP: Check controller still reconciles the TrainJob by ensuring it gets marked complete - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:536 @ 11/02/26 21:58:23.777 < Exit [It] should handle connection errors gracefully without crashing - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:505 @ 11/02/26 21:58:31.851 (10.251s) > Enter [AfterEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:499 @ 11/02/26 21:58:31.851 < Exit [AfterEach] When metrics endpoint is unreachable - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:499 @ 11/02/26 21:58:31.867 (16ms) > Enter [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:58:31.867 < Exit [AfterEach] RHAI Progression Tracking E2E Tests - /workspace/source/src_code/test/e2e/rhai/progression_e2e_test.go:86 @ 11/02/26 21:58:31.883 (16ms) > Enter [AfterSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:71 @ 11/02/26 21:58:31.883 Cleaning up test namespace: rhai-e2e-progression-sfndj < Exit [AfterSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/rhai/suite_test.go:71 @ 11/02/26 21:58:31.902 (19ms) === RUN TestKueueSchedulingReachesTrainJobPods kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK kueue_operator.go:227: Kueue CR 'cluster' exists kueue_operator.go:229: Waiting for Kueue CR to be ready... kueue_operator.go:236: Kueue CR is ready kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR... kueue_operator.go:243: TrainJob framework is present in Kueue CR trainer_kueue_integration_test.go:251: Created Kueue-managed namespace: test-ns-t762k trainer_kueue_integration_test.go:259: Created Kueue ResourceFlavor rf-vj7rd successfully trainer_kueue_integration_test.go:274: Created Kueue ClusterQueue cq-vmlds successfully trainer_kueue_integration_test.go:298: Created Kueue LocalQueue test-ns-t762k/lq-kjl47 successfully trainer_kueue_integration_test.go:332: Created TrainJob: test-runtimepatches-trainjob-hk4kj trainer_kueue_integration_test.go:340: Workload admitted trainer_kueue_integration_test.go:351: replicated job "node" should carry the ResourceFlavor's node label Expected <map[string]string | len:0>: nil to have {key: value} <map[interface {}]interface {} | len:1>: { <string>"kueue-test-flavor": <string>"on", } test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueSchedulingReachesTrainJobPods3118142704 kueue_operator.go:100: Collecting Kueue diagnostics... kueue_operator.go:100: Stored Kueue CR state kueue_operator.go:100: DataScienceCluster is not installed; skipping DSC diagnostics kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-668dc9fcbf-rkprv/manager kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-668dc9fcbf-zsqvc/manager --- FAIL: TestKueueSchedulingReachesTrainJobPods (2.22s)