=== RUN TestRunTrainJobWithDefaultClusterTrainingRuntimes cluster_training_runtimes_test.go:206: Running TrainJob with ClusterTrainingRuntime: torch-distributed cluster_training_runtimes_test.go:212: Created TrainJob test-ns-hfvc2/test-trainjob-rtnr2 successfully test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestRunTrainJobWithDefaultClusterTrainingRuntimes378623454 cluster_training_runtimes_test.go:223: TrainJob with ClusterTrainingRuntime 'torch-distributed' completed successfully dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications cluster_training_runtimes_test.go:275: No running rhods-operator pod found in redhat-ods-operator namespace test.go:171: Retrieving Pod Container test-ns-hfvc2/test-trainjob-rtnr2-node-0-0-nnj8j/node logs --- FAIL: TestRunTrainJobWithDefaultClusterTrainingRuntimes (6.30s) === RUN TestKubeflowSdkKueueIntegration kueue_operator.go:110: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster... kueue_operator.go:112: Should be able to set DSC kueue to Unmanaged Unexpected error: <*errors.StatusError | 0xb0e4d928500>: the server could not find the requested resource { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "the server could not find the requested resource", Reason: "NotFound", Details: { Name: "", Group: "", Kind: "", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred kueue_operator.go:100: Collecting Kueue diagnostics... test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration826826063 kueue_operator.go:100: Stored Kueue CR state kueue_operator.go:100: Failed to get DSC for diagnostics: the server could not find the requested resource kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-fds8l/manager kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-tt77n/manager --- FAIL: TestKubeflowSdkKueueIntegration (0.14s) === RUN TestKueueWorkloadPreemptionSuspendsTrainJob kueue_operator.go:110: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster... kueue_operator.go:112: Should be able to set DSC kueue to Unmanaged Unexpected error: <*errors.StatusError | 0xb0e4d980be0>: the server could not find the requested resource { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "the server could not find the requested resource", Reason: "NotFound", Details: { Name: "", Group: "", Kind: "", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred kueue_operator.go:100: Collecting Kueue diagnostics... test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueWorkloadPreemptionSuspendsTrainJob2549343212 kueue_operator.go:100: Stored Kueue CR state kueue_operator.go:100: Failed to get DSC for diagnostics: the server could not find the requested resource kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-fds8l/manager kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-tt77n/manager --- FAIL: TestKueueWorkloadPreemptionSuspendsTrainJob (0.13s) === RUN TestKueueSchedulingReachesTrainJobPods kueue_operator.go:110: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster... kueue_operator.go:112: Should be able to set DSC kueue to Unmanaged Unexpected error: <*errors.StatusError | 0xb0e4d6a0be0>: the server could not find the requested resource { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "the server could not find the requested resource", Reason: "NotFound", Details: { Name: "", Group: "", Kind: "", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred kueue_operator.go:100: Collecting Kueue diagnostics... test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueSchedulingReachesTrainJobPods1678157793 kueue_operator.go:100: Stored Kueue CR state kueue_operator.go:100: Failed to get DSC for diagnostics: the server could not find the requested resource kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-fds8l/manager kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-tt77n/manager --- FAIL: TestKueueSchedulingReachesTrainJobPods (0.12s)