Warning: Failed to get DSC: the server could not find the requested resource Initial Kueue managementState: === RUN TestDefaultClusterTrainingRuntimes test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Smoke' --- SKIP: TestDefaultClusterTrainingRuntimes (0.00s) === RUN TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Smoke' --- SKIP: TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes (0.00s) === RUN TestRunTrainJobWithDefaultClusterTrainingRuntimes cluster_training_runtimes_test.go:161: Running TrainJob with ClusterTrainingRuntime: torch-distributed cluster_training_runtimes_test.go:167: Created TrainJob test-ns-c27s9/test-trainjob-m6npq successfully cluster_training_runtimes_test.go:178: TrainJob with ClusterTrainingRuntime 'torch-distributed' completed successfully dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: kubeflow-trainer-system cluster_training_runtimes_test.go:161: Running TrainJob with ClusterTrainingRuntime: torch-distributed-rocm cluster_training_runtimes_test.go:167: Created TrainJob test-ns-4zv69/test-trainjob-6jc8l successfully cluster_training_runtimes_test.go:178: TrainJob with ClusterTrainingRuntime 'torch-distributed-rocm' completed successfully dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: kubeflow-trainer-system cluster_training_runtimes_test.go:161: Running TrainJob with ClusterTrainingRuntime: torch-distributed-cpu cluster_training_runtimes_test.go:167: Created TrainJob test-ns-l6xgt/test-trainjob-c6jcg successfully cluster_training_runtimes_test.go:178: TrainJob with ClusterTrainingRuntime 'torch-distributed-cpu' completed successfully dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: kubeflow-trainer-system cluster_training_runtimes_test.go:161: Running TrainJob with ClusterTrainingRuntime: torch-distributed-cuda128-torch29-py312 cluster_training_runtimes_test.go:167: Created TrainJob test-ns-snpv9/test-trainjob-jhvjt successfully cluster_training_runtimes_test.go:178: TrainJob with ClusterTrainingRuntime 'torch-distributed-cuda128-torch29-py312' completed successfully dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: kubeflow-trainer-system cluster_training_runtimes_test.go:161: Running TrainJob with ClusterTrainingRuntime: torch-distributed-rocm64-torch29-py312 cluster_training_runtimes_test.go:167: Created TrainJob test-ns-njm8z/test-trainjob-lpm72 successfully cluster_training_runtimes_test.go:178: TrainJob with ClusterTrainingRuntime 'torch-distributed-rocm64-torch29-py312' completed successfully dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: kubeflow-trainer-system cluster_training_runtimes_test.go:156: Skipping ClusterTrainingRuntime 'torch-distributed-cuda130-torch291-py312' (image 'odh-th06-cuda130-torch291-py312' already tested) cluster_training_runtimes_test.go:156: Skipping ClusterTrainingRuntime 'torch-distributed-rocm64-torch291-py312' (image 'odh-th06-rocm64-torch291-py312' already tested) cluster_training_runtimes_test.go:156: Skipping ClusterTrainingRuntime 'torch-distributed-cpu-torch291-py312' (image 'odh-th06-cpu-torch291-py312' already tested) cluster_training_runtimes_test.go:156: Skipping ClusterTrainingRuntime 'training-hub' (image 'odh-th06-cuda130-torch291-py312' already tested) cluster_training_runtimes_test.go:156: Skipping ClusterTrainingRuntime 'training-hub-cpu' (image 'odh-th06-cpu-torch291-py312' already tested) cluster_training_runtimes_test.go:156: Skipping ClusterTrainingRuntime 'training-hub-rocm' (image 'odh-th06-rocm64-torch291-py312' already tested) cluster_training_runtimes_test.go:156: Skipping ClusterTrainingRuntime 'training-hub-th05-cuda128-torch29-py312' (image 'odh-training-cuda128-torch29-py312' already tested) cluster_training_runtimes_test.go:156: Skipping ClusterTrainingRuntime 'training-hub-th06-cuda130-torch291-py312' (image 'odh-th06-cuda130-torch291-py312' already tested) cluster_training_runtimes_test.go:156: Skipping ClusterTrainingRuntime 'training-hub-th06-cpu-torch291-py312' (image 'odh-th06-cpu-torch291-py312' already tested) cluster_training_runtimes_test.go:156: Skipping ClusterTrainingRuntime 'training-hub-th06-rocm64-torch291-py312' (image 'odh-th06-rocm64-torch291-py312' already tested) cluster_training_runtimes_test.go:186: All TrainJobs with expected ClusterTrainingRuntimes completed successfully !!! test.go:169: Retrieving Pod Container test-ns-njm8z/test-trainjob-lpm72-node-0-0-dwqws/node logs test.go:152: Creating ephemeral output directory as TEST_OUTPUT_DIR env variable is unset test.go:160: Output directory has been created at: /tmp/TestRunTrainJobWithDefaultClusterTrainingRuntimes3969898063 test.go:169: Retrieving Pod Container test-ns-snpv9/test-trainjob-jhvjt-node-0-0-9c47g/node logs test.go:169: Retrieving Pod Container test-ns-l6xgt/test-trainjob-c6jcg-node-0-0-fj677/node logs test.go:169: Retrieving Pod Container test-ns-4zv69/test-trainjob-6jc8l-node-0-0-bkkpl/node logs test.go:169: Retrieving Pod Container test-ns-c27s9/test-trainjob-m6npq-node-0-0-ngvlg/node logs --- PASS: TestRunTrainJobWithDefaultClusterTrainingRuntimes (1635.11s) === RUN TestJobSetWorkflow jobset_workflow_test.go:46: Created PersistentVolumeClaim test-ns-wshsg/pvc-k5v7l successfully utils_runtimes.go:122: Using image from ClusterTrainingRuntime "torch-distributed-cpu": quay.io/opendatahub/odh-th06-cpu-torch291-py312@sha256:6a2e7309dbb50d1a3e6e0033aa92ffc8dde7a12ab89798a5b59d5f2fec010978 jobset_workflow_test.go:53: Created TrainingRuntime test-ns-wshsg/test-trainingruntime-hg8ms jobset_workflow_test.go:56: Created TrainJob test-ns-wshsg/test-trainjob-jnd7b jobset_workflow_test.go:62: JobSet created with 3 replicated jobs (dataset-initializer, model-initializer, node) jobset_workflow_test.go:65: Monitoring job execution ... jobset_workflow_test.go:65: dataset-initializer job is created: test-trainjob-jnd7b-dataset-initializer-0 jobset_workflow_test.go:65: dataset-initializer job is completed: test-trainjob-jnd7b-dataset-initializer-0 jobset_workflow_test.go:65: model-initializer job is created: test-trainjob-jnd7b-model-initializer-0 jobset_workflow_test.go:65: model-initializer job is completed: test-trainjob-jnd7b-model-initializer-0 jobset_workflow_test.go:65: node job is created: test-trainjob-jnd7b-node-0 jobset_workflow_test.go:65: Sequential job execution is verified successfully: dataset-initializer → model-initializer → node jobset_workflow_test.go:70: TrainJob test-ns-wshsg/test-trainjob-jnd7b completed test.go:169: Retrieving Pod Container test-ns-wshsg/test-trainjob-jnd7b-dataset-initializer-0-0-df8d2/dataset-initializer logs test.go:152: Creating ephemeral output directory as TEST_OUTPUT_DIR env variable is unset test.go:160: Output directory has been created at: /tmp/TestJobSetWorkflow202192770 test.go:169: Retrieving Pod Container test-ns-wshsg/test-trainjob-jnd7b-model-initializer-0-0-xxlpl/model-initializer logs test.go:169: Retrieving Pod Container test-ns-wshsg/test-trainjob-jnd7b-node-0-0-ssss9/node logs --- PASS: TestJobSetWorkflow (125.99s) === RUN TestFailedJobSetWorkflow jobset_workflow_test.go:81: Created PersistentVolumeClaim test-ns-2whsd/pvc-kmhd9 successfully utils_runtimes.go:122: Using image from ClusterTrainingRuntime "torch-distributed-cpu": quay.io/opendatahub/odh-th06-cpu-torch291-py312@sha256:6a2e7309dbb50d1a3e6e0033aa92ffc8dde7a12ab89798a5b59d5f2fec010978 jobset_workflow_test.go:88: Created TrainingRuntime test-ns-2whsd/test-trainingruntime-d25zf jobset_workflow_test.go:91: Created TrainJob test-ns-2whsd/test-trainjob-fail-gjjds jobset_workflow_test.go:100: JobSet failed as expected jobset_workflow_test.go:105: TrainJob failed as expected test.go:169: Retrieving Pod Container test-ns-2whsd/test-trainjob-fail-gjjds-dataset-initializer-0-0-5clzg/dataset-initializer logs test.go:152: Creating ephemeral output directory as TEST_OUTPUT_DIR env variable is unset test.go:160: Output directory has been created at: /tmp/TestFailedJobSetWorkflow2645360101 --- PASS: TestFailedJobSetWorkflow (11.16s) === RUN TestKubeflowSdkSanity environment.go:75: Expected environment variable NOTEBOOK_USER_NAME not found, please use this environment variable to specify name of the authenticated Notebook user. test.go:152: Creating ephemeral output directory as TEST_OUTPUT_DIR env variable is unset test.go:160: Output directory has been created at: /tmp/TestKubeflowSdkSanity192669226 --- FAIL: TestKubeflowSdkSanity (0.10s) === RUN TestKubeflowSdkKueueIntegration kueue_operator.go:99: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster... kueue_operator.go:101: Should be able to set DSC kueue to Unmanaged Unexpected error: <*errors.StatusError | 0xc0009c21e0>: the server could not find the requested resource { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "the server could not find the requested resource", Reason: "NotFound", Details: { Name: "", Group: "", Kind: "", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred --- FAIL: TestKubeflowSdkKueueIntegration (0.00s) === RUN TestSftTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestSftTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestOsftTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestOsftTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestLoraTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestLoraTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestOsftTrainingHubMultiNodeMultiGPU test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestOsftTrainingHubMultiNodeMultiGPU (0.00s) === RUN TestLoraTrainingHubMultiNodeMultiGPU test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestLoraTrainingHubMultiNodeMultiGPU (0.00s) === RUN TestSftTrainingHubMultiNodeMultiGPU test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestSftTrainingHubMultiNodeMultiGPU (0.00s) === RUN TestRhaiTrainingProgressionCPU test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Tier1' --- SKIP: TestRhaiTrainingProgressionCPU (0.00s) === RUN TestRhaiJitCheckpointingCPU test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Tier1' --- SKIP: TestRhaiJitCheckpointingCPU (0.00s) === RUN TestRhaiFeaturesCPU test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Tier1' --- SKIP: TestRhaiFeaturesCPU (0.00s) === RUN TestRhaiTrainingProgressionCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiTrainingProgressionCuda (0.00s) === RUN TestRhaiJitCheckpointingCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiJitCheckpointingCuda (0.00s) === RUN TestRhaiFeaturesCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiFeaturesCuda (0.00s) === RUN TestRhaiTrainingProgressionRocm test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiTrainingProgressionRocm (0.00s) === RUN TestRhaiJitCheckpointingRocm test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiJitCheckpointingRocm (0.00s) === RUN TestRhaiFeaturesRocm test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiFeaturesRocm (0.00s) === RUN TestRhaiTrainingProgressionMultiGpuCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiTrainingProgressionMultiGpuCuda (0.00s) === RUN TestRhaiJitCheckpointingMultiGpuCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiJitCheckpointingMultiGpuCuda (0.00s) === RUN TestRhaiFeaturesMultiGpuCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiFeaturesMultiGpuCuda (0.00s) === RUN TestRhaiTrainingProgressionMultiGpuRocm test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiTrainingProgressionMultiGpuRocm (0.00s) === RUN TestRhaiJitCheckpointingMultiGpuRocm test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiJitCheckpointingMultiGpuRocm (0.00s) === RUN TestRhaiFeaturesMultiGpuRocm test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiFeaturesMultiGpuRocm (0.00s) === RUN TestTrainingFailureScenarios test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestTrainingFailureScenarios (0.00s) === RUN TestTorchrunTrainingFailure test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestTorchrunTrainingFailure (0.00s) === RUN TestRhaiS3CheckpointingCPU test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Tier1' --- SKIP: TestRhaiS3CheckpointingCPU (0.00s) === RUN TestRhaiS3FsdpFullStateCheckpointingCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3FsdpFullStateCheckpointingCuda (0.00s) === RUN TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda (0.00s) === RUN TestRhaiS3FsdpSharedStateCheckpointingCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3FsdpSharedStateCheckpointingCuda (0.00s) === RUN TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda (0.00s) === RUN TestRhaiS3DeepspeedStage0CheckpointingCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3DeepspeedStage0CheckpointingCuda (0.00s) === RUN TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda (0.00s) === RUN TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Tier1' --- SKIP: TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 (0.00s) === RUN TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda (0.00s) === RUN TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda (0.00s) === RUN TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda (0.00s) === RUN TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda (0.00s) === RUN TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm (0.00s) === RUN TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm (0.00s) === RUN TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm (0.00s) === RUN TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm (0.00s) === RUN TestKueueDefaultLocalQueueLabelInjection kueue_operator.go:99: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster... kueue_operator.go:101: Should be able to set DSC kueue to Unmanaged Unexpected error: <*errors.StatusError | 0xc000476000>: the server could not find the requested resource { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "the server could not find the requested resource", Reason: "NotFound", Details: { Name: "", Group: "", Kind: "", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred --- FAIL: TestKueueDefaultLocalQueueLabelInjection (0.00s) === RUN TestKueueWorkloadPreemptionSuspendsTrainJob kueue_operator.go:99: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster... kueue_operator.go:101: Should be able to set DSC kueue to Unmanaged Unexpected error: <*errors.StatusError | 0xc0006da460>: the server could not find the requested resource { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "the server could not find the requested resource", Reason: "NotFound", Details: { Name: "", Group: "", Kind: "", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred --- FAIL: TestKueueWorkloadPreemptionSuspendsTrainJob (0.00s) === RUN TestKueueWorkloadInadmissibleWithNonExistentLocalQueue kueue_operator.go:99: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster... kueue_operator.go:101: Should be able to set DSC kueue to Unmanaged Unexpected error: <*errors.StatusError | 0xc0007df680>: the server could not find the requested resource { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "the server could not find the requested resource", Reason: "NotFound", Details: { Name: "", Group: "", Kind: "", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred --- FAIL: TestKueueWorkloadInadmissibleWithNonExistentLocalQueue (0.00s) === RUN TestSetupUpgradeTrainJob trainer_kueue_upgrade_training_test.go:57: Skip due to issue RHOAIENG-48867 --- SKIP: TestSetupUpgradeTrainJob (0.00s) === RUN TestRunUpgradeTrainJob trainer_kueue_upgrade_training_test.go:125: Skip due to issue RHOAIENG-48867 --- SKIP: TestRunUpgradeTrainJob (0.00s) === RUN TestSetupSpecificRuntimeUpgradeTrainJob test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Pre-Upgrade' --- SKIP: TestSetupSpecificRuntimeUpgradeTrainJob (0.00s) === RUN TestRunSpecificRuntimeUpgradeTrainJob test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Post-Upgrade' --- SKIP: TestRunSpecificRuntimeUpgradeTrainJob (0.00s) === RUN TestKubeflowTrainerSmoke test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Smoke' --- SKIP: TestKubeflowTrainerSmoke (0.00s) === RUN TestSetupTrainingRuntime test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Pre-Upgrade' --- SKIP: TestSetupTrainingRuntime (0.00s) === RUN TestVerifyTrainingRuntime test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Post-Upgrade' --- SKIP: TestVerifyTrainingRuntime (0.00s) === RUN TestSetupSleepTrainJob test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Pre-Upgrade' --- SKIP: TestSetupSleepTrainJob (0.00s) === RUN TestVerifySleepTrainJob test_tag.go:37: Test tier 'Sanity' doesn't match expected tier 'Post-Upgrade' --- SKIP: TestVerifySleepTrainJob (0.00s) FAIL TearDown: Setting kueue managementState to Removed in DataScienceCluster... TearDown: Failed to set Kueue to Removed: TearDown: failed to set kueue to Removed: the server could not find the requested resource ok github.com/opendatahub-io/distributed-workloads/tests/trainer 1772.447s