> Enter [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:42 @ 10/22/26 17:07:23.708 < Exit [BeforeSuite] TOP-LEVEL - /workspace/source/src_code/test/e2e/suite_test.go:42 @ 10/22/26 17:07:23.709 (1ms) [FAILED] Timed out after 600.001s. The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:84 with: Expected object to be comparable, diff: []v1.Condition( - nil, + { + { + Type: "Complete", + Status: "True", + Reason: "AllJobsCompleted", + Message: "jobset completed successfully", + }, + }, ) In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:102 @ 10/22/26 17:17:24.453 > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 10/22/26 17:07:23.709 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 10/22/26 17:07:23.834 (125ms) > Enter [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:51 @ 10/22/26 17:07:23.834 STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:57 @ 10/22/26 17:07:23.834 END STEP: Create a TrainJob with torch-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:57 @ 10/22/26 17:07:23.915 (81ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:62 @ 10/22/26 17:07:23.915 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:62 @ 10/22/26 17:07:24.452 (537ms) STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:80 @ 10/22/26 17:07:24.452 END STEP: Wait for TrainJob to be in Succeeded status with all jobs succeeded - /workspace/source/src_code/test/e2e/e2e_test.go:80 @ 10/22/26 17:17:24.453 (10m0.001s) [FAILED] Timed out after 600.001s. The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:84 with: Expected object to be comparable, diff: []v1.Condition( - nil, + { + { + Type: "Complete", + Status: "True", + Reason: "AllJobsCompleted", + Message: "jobset completed successfully", + }, + }, ) In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:102 @ 10/22/26 17:17:24.453 < Exit [It] should create TrainJob with PyTorch runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:51 @ 10/22/26 17:17:24.454 (10m0.619s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 10/22/26 17:17:24.454 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 10/22/26 17:17:24.467 (14ms) [FAILED] Timed out after 600.000s. The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:124 with: Expected object to be comparable, diff: []v1alpha1.JobStatus(Inverse(cmpopts.SortSlices, []v1alpha1.JobStatus{ { ... // 2 identical fields Succeeded: &0, Failed: &0, - Active: &0, + Active: &1, Suspended: &0, }, { ... // 2 identical fields Succeeded: &0, Failed: &0, - Active: &0, + Active: &1, Suspended: &0, }, })) In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:142 @ 10/22/26 17:27:24.576 > Enter [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 10/22/26 17:17:24.467 < Exit [BeforeEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:28 @ 10/22/26 17:17:24.495 (28ms) > Enter [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:109 @ 10/22/26 17:17:24.495 STEP: Create a TrainJob with deepspeed-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:115 @ 10/22/26 17:17:24.495 END STEP: Create a TrainJob with deepspeed-distributed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:115 @ 10/22/26 17:17:24.575 (80ms) STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:120 @ 10/22/26 17:17:24.575 END STEP: Wait for TrainJob jobs to become active - /workspace/source/src_code/test/e2e/e2e_test.go:120 @ 10/22/26 17:27:24.576 (10m0.001s) [FAILED] Timed out after 600.000s. The function passed to Eventually failed at /workspace/source/src_code/test/e2e/e2e_test.go:124 with: Expected object to be comparable, diff: []v1alpha1.JobStatus(Inverse(cmpopts.SortSlices, []v1alpha1.JobStatus{ { ... // 2 identical fields Succeeded: &0, Failed: &0, - Active: &0, + Active: &1, Suspended: &0, }, { ... // 2 identical fields Succeeded: &0, Failed: &0, - Active: &0, + Active: &1, Suspended: &0, }, })) In [It] at: /workspace/source/src_code/test/e2e/e2e_test.go:142 @ 10/22/26 17:27:24.576 < Exit [It] should create TrainJob with DeepSpeed runtime reference - /workspace/source/src_code/test/e2e/e2e_test.go:109 @ 10/22/26 17:27:24.576 (10m0.081s) > Enter [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 10/22/26 17:27:24.576 < Exit [AfterEach] TrainJob e2e - /workspace/source/src_code/test/e2e/e2e_test.go:43 @ 10/22/26 17:27:24.592 (16ms) === RUN TestRunTrainJobWithDefaultClusterTrainingRuntimes cluster_training_runtimes_test.go:206: Running TrainJob with ClusterTrainingRuntime: torch-distributed cluster_training_runtimes_test.go:212: Created TrainJob test-ns-hfvc2/test-trainjob-rtnr2 successfully test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestRunTrainJobWithDefaultClusterTrainingRuntimes378623454 cluster_training_runtimes_test.go:223: TrainJob with ClusterTrainingRuntime 'torch-distributed' completed successfully dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications cluster_training_runtimes_test.go:275: No running rhods-operator pod found in redhat-ods-operator namespace test.go:171: Retrieving Pod Container test-ns-hfvc2/test-trainjob-rtnr2-node-0-0-nnj8j/node logs --- FAIL: TestRunTrainJobWithDefaultClusterTrainingRuntimes (6.30s) === RUN TestKubeflowSdkKueueIntegration kueue_operator.go:110: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster... kueue_operator.go:112: Should be able to set DSC kueue to Unmanaged Unexpected error: <*errors.StatusError | 0xb0e4d928500>: the server could not find the requested resource { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "the server could not find the requested resource", Reason: "NotFound", Details: { Name: "", Group: "", Kind: "", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred kueue_operator.go:100: Collecting Kueue diagnostics... test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration826826063 kueue_operator.go:100: Stored Kueue CR state kueue_operator.go:100: Failed to get DSC for diagnostics: the server could not find the requested resource kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-fds8l/manager kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-tt77n/manager --- FAIL: TestKubeflowSdkKueueIntegration (0.14s) === RUN TestKueueWorkloadPreemptionSuspendsTrainJob kueue_operator.go:110: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster... kueue_operator.go:112: Should be able to set DSC kueue to Unmanaged Unexpected error: <*errors.StatusError | 0xb0e4d980be0>: the server could not find the requested resource { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "the server could not find the requested resource", Reason: "NotFound", Details: { Name: "", Group: "", Kind: "", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred kueue_operator.go:100: Collecting Kueue diagnostics... test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueWorkloadPreemptionSuspendsTrainJob2549343212 kueue_operator.go:100: Stored Kueue CR state kueue_operator.go:100: Failed to get DSC for diagnostics: the server could not find the requested resource kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-fds8l/manager kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-tt77n/manager --- FAIL: TestKueueWorkloadPreemptionSuspendsTrainJob (0.13s) === RUN TestKueueSchedulingReachesTrainJobPods kueue_operator.go:110: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster... kueue_operator.go:112: Should be able to set DSC kueue to Unmanaged Unexpected error: <*errors.StatusError | 0xb0e4d6a0be0>: the server could not find the requested resource { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "the server could not find the requested resource", Reason: "NotFound", Details: { Name: "", Group: "", Kind: "", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred kueue_operator.go:100: Collecting Kueue diagnostics... test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueSchedulingReachesTrainJobPods1678157793 kueue_operator.go:100: Stored Kueue CR state kueue_operator.go:100: Failed to get DSC for diagnostics: the server could not find the requested resource kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-fds8l/manager kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-tt77n/manager --- FAIL: TestKueueSchedulingReachesTrainJobPods (0.12s)