Warning: Failed to get DSC: the server could not find the requested resource
Initial Kueue managementState: 
=== RUN   TestDefaultClusterTrainingRuntimes
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'
--- SKIP: TestDefaultClusterTrainingRuntimes (0.00s)
=== RUN   TestOpenMPICudaClusterTrainingRuntime
    cluster_training_runtimes_test.go:120: Skip due to issue RHOAIENG-61966
--- SKIP: TestOpenMPICudaClusterTrainingRuntime (0.00s)
=== RUN   TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'
--- SKIP: TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes (0.00s)
=== RUN   TestRunTrainJobWithDefaultClusterTrainingRuntimes
    cluster_training_runtimes_test.go:206: Running TrainJob with ClusterTrainingRuntime: torch-distributed
    cluster_training_runtimes_test.go:212: Created TrainJob test-ns-hfvc2/test-trainjob-rtnr2 successfully
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestRunTrainJobWithDefaultClusterTrainingRuntimes378623454
    cluster_training_runtimes_test.go:223: TrainJob with ClusterTrainingRuntime 'torch-distributed' completed successfully
    dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications
    cluster_training_runtimes_test.go:275: No running rhods-operator pod found in redhat-ods-operator namespace
    test.go:171: Retrieving Pod Container test-ns-hfvc2/test-trainjob-rtnr2-node-0-0-nnj8j/node logs
--- FAIL: TestRunTrainJobWithDefaultClusterTrainingRuntimes (6.30s)
=== RUN   TestJobSetWorkflow
    jobset_workflow_test.go:45: Created PersistentVolumeClaim test-ns-7fqpz/pvc-766bk successfully
    utils_runtimes.go:136: Using image from ClusterTrainingRuntime "torch-distributed-cpu": quay.io/opendatahub/odh-th-torch-cpu-py312:odh-3.5
    jobset_workflow_test.go:52: Created TrainingRuntime test-ns-7fqpz/test-trainingruntime-jhbln
    jobset_workflow_test.go:55: Created TrainJob test-ns-7fqpz/test-trainjob-xcb5k
    jobset_workflow_test.go:61: JobSet created with 3 replicated jobs (dataset-initializer, model-initializer, node)
    jobset_workflow_test.go:64: Monitoring job execution ...
    jobset_workflow_test.go:64: dataset-initializer job is created: test-trainjob-xcb5k-dataset-initializer-0
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestJobSetWorkflow1571147198
    jobset_workflow_test.go:64: dataset-initializer job is completed: test-trainjob-xcb5k-dataset-initializer-0
    jobset_workflow_test.go:64: model-initializer job is created: test-trainjob-xcb5k-model-initializer-0
    jobset_workflow_test.go:64: model-initializer job is completed: test-trainjob-xcb5k-model-initializer-0
    jobset_workflow_test.go:64: node job is created: test-trainjob-xcb5k-node-0
    jobset_workflow_test.go:64: Sequential job execution is verified successfully: dataset-initializer → model-initializer → node
    jobset_workflow_test.go:69: TrainJob test-ns-7fqpz/test-trainjob-xcb5k completed
    test.go:171: Retrieving Pod Container test-ns-7fqpz/test-trainjob-xcb5k-dataset-initializer-0-0-78fvl/dataset-initializer logs
    test.go:171: Retrieving Pod Container test-ns-7fqpz/test-trainjob-xcb5k-model-initializer-0-0-fsl5w/model-initializer logs
    test.go:171: Retrieving Pod Container test-ns-7fqpz/test-trainjob-xcb5k-node-0-0-qdv4p/node logs
--- PASS: TestJobSetWorkflow (176.85s)
=== RUN   TestFailedJobSetWorkflow
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestFailedJobSetWorkflow (0.00s)
=== RUN   TestKubeflowSdk
    fashion_mnist_tests.go:54: Created User RoleBinding test-ns-wk5xf/rb-hg9cs successfully
    fashion_mnist_tests.go:55: Created User RoleBinding test-ns-wk5xf/rb-dr958 successfully
    fashion_mnist_tests.go:55: Created ClusterRole /clusterrole-6hgm2 successfully
    fashion_mnist_tests.go:55: Created User ClusterRoleBinding crb-sfm94 successfully
    fashion_mnist_tests.go:62: Created ConfigMap test-ns-wk5xf/config-tktfq successfully
    fashion_mnist_tests.go:80: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX.
    fashion_mnist_tests.go:82: Created PersistentVolumeClaim test-ns-wk5xf/pvc-t6dmh successfully
    fashion_mnist_tests.go:91: OpenShift API URL: https://ae29190c5bc7746cb8514735bcacd07d-4004b3586869de14.elb.us-east-2.amazonaws.com:6443
    dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications
    environment.go:164: Using notebook image from ImageStream training-hub-universal-cpu:3.5: quay.io/opendatahub/odh-th-torch-cpu-py312@sha256:28eb0a744e9b3f0696d31cffa718c0e478aca37f46503840e52ce0bd3ca56469
    notebook.go:44: Created test Deployment test-ns-wk5xf/dw-deployment-bvk5k
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdk607326528
    test.go:171: Retrieving Pod Container test-ns-wk5xf/dw-deployment-bvk5k-55d99d69bd-b2xfd/notebook logs
--- PASS: TestKubeflowSdk (114.72s)
=== RUN   TestKubeflowSdkKueueIntegration
    kueue_operator.go:110: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster...
    kueue_operator.go:112: 
        Should be able to set DSC kueue to Unmanaged
        Unexpected error:
            <*errors.StatusError | 0xb0e4d928500>: 
            the server could not find the requested resource
            {
                ErrStatus: {
                    TypeMeta: {Kind: "", APIVersion: ""},
                    ListMeta: {
                        SelfLink: "",
                        ResourceVersion: "",
                        Continue: "",
                        RemainingItemCount: nil,
                    },
                    Status: "Failure",
                    Message: "the server could not find the requested resource",
                    Reason: "NotFound",
                    Details: {
                        Name: "",
                        Group: "",
                        Kind: "",
                        UID: "",
                        Causes: [
                            {
                                Type: "UnexpectedServerResponse",
                                Message: "404 page not found",
                                Field: "",
                            },
                        ],
                        RetryAfterSeconds: 0,
                    },
                    Code: 404,
                },
            }
        occurred
    kueue_operator.go:100: Collecting Kueue diagnostics...
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration826826063
    kueue_operator.go:100: Stored Kueue CR state
    kueue_operator.go:100: Failed to get DSC for diagnostics: the server could not find the requested resource
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-fds8l/manager
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-tt77n/manager
--- FAIL: TestKubeflowSdkKueueIntegration (0.14s)
=== RUN   TestKubeflowSdkOpenMPICudaKueueIntegration
    kubeflow_sdk_test.go:40: Skip due to issue RHOAIENG-61966
--- SKIP: TestKubeflowSdkOpenMPICudaKueueIntegration (0.00s)
=== RUN   TestSftTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestSftTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestOsftTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestOsftTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestLoraTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestLoraTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestOsftTrainingHubMultiNodeMultiGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestOsftTrainingHubMultiNodeMultiGPU (0.00s)
=== RUN   TestLoraTrainingHubMultiNodeMultiGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestLoraTrainingHubMultiNodeMultiGPU (0.00s)
=== RUN   TestSftTrainingHubMultiNodeMultiGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestSftTrainingHubMultiNodeMultiGPU (0.00s)
=== RUN   TestRhaiTrainingProgressionCPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestRhaiTrainingProgressionCPU (0.00s)
=== RUN   TestRhaiJitCheckpointingCPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestRhaiJitCheckpointingCPU (0.00s)
=== RUN   TestRhaiFeaturesCPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestRhaiFeaturesCPU (0.00s)
=== RUN   TestRhaiTrainingProgressionCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiTrainingProgressionCuda (0.00s)
=== RUN   TestRhaiJitCheckpointingCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiJitCheckpointingCuda (0.00s)
=== RUN   TestRhaiFeaturesCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiFeaturesCuda (0.00s)
=== RUN   TestRhaiTrainingProgressionRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiTrainingProgressionRocm (0.00s)
=== RUN   TestRhaiJitCheckpointingRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiJitCheckpointingRocm (0.00s)
=== RUN   TestRhaiFeaturesRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiFeaturesRocm (0.00s)
=== RUN   TestRhaiTrainingProgressionMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiTrainingProgressionMultiGpuCuda (0.00s)
=== RUN   TestRhaiJitCheckpointingMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiJitCheckpointingMultiGpuCuda (0.00s)
=== RUN   TestRhaiFeaturesMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiFeaturesMultiGpuCuda (0.00s)
=== RUN   TestRhaiTrainingProgressionMultiGpuRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiTrainingProgressionMultiGpuRocm (0.00s)
=== RUN   TestRhaiJitCheckpointingMultiGpuRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiJitCheckpointingMultiGpuRocm (0.00s)
=== RUN   TestRhaiFeaturesMultiGpuRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiFeaturesMultiGpuRocm (0.00s)
=== RUN   TestTrainingFailureScenarios
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestTrainingFailureScenarios (0.00s)
=== RUN   TestTorchrunTrainingFailure
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestTorchrunTrainingFailure (0.00s)
=== RUN   TestRhaiS3CheckpointingCPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestRhaiS3CheckpointingCPU (0.00s)
=== RUN   TestRhaiS3FsdpFullStateCheckpointingCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3FsdpFullStateCheckpointingCuda (0.00s)
=== RUN   TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda (0.00s)
=== RUN   TestRhaiS3FsdpSharedStateCheckpointingCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3FsdpSharedStateCheckpointingCuda (0.00s)
=== RUN   TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda (0.00s)
=== RUN   TestRhaiS3DeepspeedStage0CheckpointingCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3DeepspeedStage0CheckpointingCuda (0.00s)
=== RUN   TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda (0.00s)
=== RUN   TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 (0.00s)
=== RUN   TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda (0.00s)
=== RUN   TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda (0.00s)
=== RUN   TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda (0.00s)
=== RUN   TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda (0.00s)
=== RUN   TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm (0.00s)
=== RUN   TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm (0.00s)
=== RUN   TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm (0.00s)
=== RUN   TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm (0.00s)
=== RUN   TestKueueWorkloadPreemptionSuspendsTrainJob
    kueue_operator.go:110: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster...
    kueue_operator.go:112: 
        Should be able to set DSC kueue to Unmanaged
        Unexpected error:
            <*errors.StatusError | 0xb0e4d980be0>: 
            the server could not find the requested resource
            {
                ErrStatus: {
                    TypeMeta: {Kind: "", APIVersion: ""},
                    ListMeta: {
                        SelfLink: "",
                        ResourceVersion: "",
                        Continue: "",
                        RemainingItemCount: nil,
                    },
                    Status: "Failure",
                    Message: "the server could not find the requested resource",
                    Reason: "NotFound",
                    Details: {
                        Name: "",
                        Group: "",
                        Kind: "",
                        UID: "",
                        Causes: [
                            {
                                Type: "UnexpectedServerResponse",
                                Message: "404 page not found",
                                Field: "",
                            },
                        ],
                        RetryAfterSeconds: 0,
                    },
                    Code: 404,
                },
            }
        occurred
    kueue_operator.go:100: Collecting Kueue diagnostics...
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueWorkloadPreemptionSuspendsTrainJob2549343212
    kueue_operator.go:100: Stored Kueue CR state
    kueue_operator.go:100: Failed to get DSC for diagnostics: the server could not find the requested resource
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-fds8l/manager
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-tt77n/manager
--- FAIL: TestKueueWorkloadPreemptionSuspendsTrainJob (0.13s)
=== RUN   TestKueueWorkloadInadmissibleWithNonExistentLocalQueue
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestKueueWorkloadInadmissibleWithNonExistentLocalQueue (0.00s)
=== RUN   TestKueueSchedulingReachesTrainJobPods
    kueue_operator.go:110: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster...
    kueue_operator.go:112: 
        Should be able to set DSC kueue to Unmanaged
        Unexpected error:
            <*errors.StatusError | 0xb0e4d6a0be0>: 
            the server could not find the requested resource
            {
                ErrStatus: {
                    TypeMeta: {Kind: "", APIVersion: ""},
                    ListMeta: {
                        SelfLink: "",
                        ResourceVersion: "",
                        Continue: "",
                        RemainingItemCount: nil,
                    },
                    Status: "Failure",
                    Message: "the server could not find the requested resource",
                    Reason: "NotFound",
                    Details: {
                        Name: "",
                        Group: "",
                        Kind: "",
                        UID: "",
                        Causes: [
                            {
                                Type: "UnexpectedServerResponse",
                                Message: "404 page not found",
                                Field: "",
                            },
                        ],
                        RetryAfterSeconds: 0,
                    },
                    Code: 404,
                },
            }
        occurred
    kueue_operator.go:100: Collecting Kueue diagnostics...
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueSchedulingReachesTrainJobPods1678157793
    kueue_operator.go:100: Stored Kueue CR state
    kueue_operator.go:100: Failed to get DSC for diagnostics: the server could not find the requested resource
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-fds8l/manager
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-tt77n/manager
--- FAIL: TestKueueSchedulingReachesTrainJobPods (0.12s)
=== RUN   TestSetupUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:82: Skip due to issue RHOAIENG-48867
--- SKIP: TestSetupUpgradeTrainJob (0.00s)
=== RUN   TestRunUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:157: Skip due to issue RHOAIENG-48867
--- SKIP: TestRunUpgradeTrainJob (0.00s)
=== RUN   TestSetupSpecificRuntimeUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:204: Skip due to issue RHOAIENG-48867
--- SKIP: TestSetupSpecificRuntimeUpgradeTrainJob (0.00s)
=== RUN   TestRunSpecificRuntimeUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:294: Skip due to issue RHOAIENG-48867
--- SKIP: TestRunSpecificRuntimeUpgradeTrainJob (0.00s)
=== RUN   TestSetupCustomRuntimeUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:372: Skip due to issue RHOAIENG-87834
--- SKIP: TestSetupCustomRuntimeUpgradeTrainJob (0.00s)
=== RUN   TestRunCustomRuntimeUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:458: Skip due to issue RHOAIENG-87834
--- SKIP: TestRunCustomRuntimeUpgradeTrainJob (0.00s)
=== RUN   TestMultiNodeOpenMPITrainJob
    trainer_mpi_test.go:35: Skip due to issue RHOAIENG-61966
--- SKIP: TestMultiNodeOpenMPITrainJob (0.00s)
=== RUN   TestOpenMPICudaTrainJobKueueIntegration
    trainer_openmpi_kueue_integration_test.go:36: Skip due to issue RHOAIENG-61966
--- SKIP: TestOpenMPICudaTrainJobKueueIntegration (0.00s)
=== RUN   TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate
    trainer_openmpi_kueue_integration_test.go:138: Skip due to issue RHOAIENG-61966
--- SKIP: TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate (0.00s)
=== RUN   TestSftStockTrlSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestSftStockTrlSingleNodeSingleGPU (0.00s)
=== RUN   TestSftStockTrlSingleNodeSingleGPUWithTorchRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestSftStockTrlSingleNodeSingleGPUWithTorchRocm (0.00s)
=== RUN   TestKubeflowTrainerSmoke
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'
--- SKIP: TestKubeflowTrainerSmoke (0.00s)
=== RUN   TestSetupTrainingRuntime
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade'
--- SKIP: TestSetupTrainingRuntime (0.00s)
=== RUN   TestVerifyTrainingRuntime
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade'
--- SKIP: TestVerifyTrainingRuntime (0.00s)
=== RUN   TestSetupSleepTrainJob
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade'
--- SKIP: TestSetupSleepTrainJob (0.00s)
=== RUN   TestVerifySleepTrainJob
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade'
--- SKIP: TestVerifySleepTrainJob (0.00s)
FAIL
TearDown: Setting kueue managementState to Removed in DataScienceCluster...
TearDown: Failed to set Kueue to Removed: TearDown: failed to set kueue to Removed: the server could not find the requested resource
ok  	github.com/opendatahub-io/distributed-workloads/tests/trainer	298.385s

=== Skipped
=== SKIP: tests/trainer TestDefaultClusterTrainingRuntimes (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'

=== SKIP: tests/trainer TestOpenMPICudaClusterTrainingRuntime (0.00s)
    cluster_training_runtimes_test.go:120: Skip due to issue RHOAIENG-61966

=== SKIP: tests/trainer TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'

=== SKIP: tests/trainer TestFailedJobSetWorkflow (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestKubeflowSdkOpenMPICudaKueueIntegration (0.00s)
    kubeflow_sdk_test.go:40: Skip due to issue RHOAIENG-61966

=== SKIP: tests/trainer TestSftTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestOsftTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestLoraTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestOsftTrainingHubMultiNodeMultiGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestLoraTrainingHubMultiNodeMultiGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestSftTrainingHubMultiNodeMultiGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiTrainingProgressionCPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestRhaiJitCheckpointingCPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestRhaiFeaturesCPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestRhaiTrainingProgressionCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiJitCheckpointingCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiFeaturesCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiTrainingProgressionRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiJitCheckpointingRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiFeaturesRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiTrainingProgressionMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiJitCheckpointingMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiFeaturesMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiTrainingProgressionMultiGpuRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiJitCheckpointingMultiGpuRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiFeaturesMultiGpuRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestTrainingFailureScenarios (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestTorchrunTrainingFailure (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3CheckpointingCPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestRhaiS3FsdpFullStateCheckpointingCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3FsdpSharedStateCheckpointingCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3DeepspeedStage0CheckpointingCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestKueueWorkloadInadmissibleWithNonExistentLocalQueue (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestSetupUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:82: Skip due to issue RHOAIENG-48867

=== SKIP: tests/trainer TestRunUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:157: Skip due to issue RHOAIENG-48867

=== SKIP: tests/trainer TestSetupSpecificRuntimeUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:204: Skip due to issue RHOAIENG-48867

=== SKIP: tests/trainer TestRunSpecificRuntimeUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:294: Skip due to issue RHOAIENG-48867

=== SKIP: tests/trainer TestSetupCustomRuntimeUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:372: Skip due to issue RHOAIENG-87834

=== SKIP: tests/trainer TestRunCustomRuntimeUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:458: Skip due to issue RHOAIENG-87834

=== SKIP: tests/trainer TestMultiNodeOpenMPITrainJob (0.00s)
    trainer_mpi_test.go:35: Skip due to issue RHOAIENG-61966

=== SKIP: tests/trainer TestOpenMPICudaTrainJobKueueIntegration (0.00s)
    trainer_openmpi_kueue_integration_test.go:36: Skip due to issue RHOAIENG-61966

=== SKIP: tests/trainer TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate (0.00s)
    trainer_openmpi_kueue_integration_test.go:138: Skip due to issue RHOAIENG-61966

=== SKIP: tests/trainer TestSftStockTrlSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestSftStockTrlSingleNodeSingleGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestKubeflowTrainerSmoke (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'

=== SKIP: tests/trainer TestSetupTrainingRuntime (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade'

=== SKIP: tests/trainer TestVerifyTrainingRuntime (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade'

=== SKIP: tests/trainer TestSetupSleepTrainJob (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade'

=== SKIP: tests/trainer TestVerifySleepTrainJob (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade'

=== Failed
=== FAIL: tests/trainer TestRunTrainJobWithDefaultClusterTrainingRuntimes (6.30s)
    cluster_training_runtimes_test.go:206: Running TrainJob with ClusterTrainingRuntime: torch-distributed
    cluster_training_runtimes_test.go:212: Created TrainJob test-ns-hfvc2/test-trainjob-rtnr2 successfully
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestRunTrainJobWithDefaultClusterTrainingRuntimes378623454
    cluster_training_runtimes_test.go:223: TrainJob with ClusterTrainingRuntime 'torch-distributed' completed successfully
    dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications
    cluster_training_runtimes_test.go:275: No running rhods-operator pod found in redhat-ods-operator namespace
    test.go:171: Retrieving Pod Container test-ns-hfvc2/test-trainjob-rtnr2-node-0-0-nnj8j/node logs

=== FAIL: tests/trainer TestKubeflowSdkKueueIntegration (0.14s)
    kueue_operator.go:110: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster...
    kueue_operator.go:112: 
        Should be able to set DSC kueue to Unmanaged
        Unexpected error:
            <*errors.StatusError | 0xb0e4d928500>: 
            the server could not find the requested resource
            {
                ErrStatus: {
                    TypeMeta: {Kind: "", APIVersion: ""},
                    ListMeta: {
                        SelfLink: "",
                        ResourceVersion: "",
                        Continue: "",
                        RemainingItemCount: nil,
                    },
                    Status: "Failure",
                    Message: "the server could not find the requested resource",
                    Reason: "NotFound",
                    Details: {
                        Name: "",
                        Group: "",
                        Kind: "",
                        UID: "",
                        Causes: [
                            {
                                Type: "UnexpectedServerResponse",
                                Message: "404 page not found",
                                Field: "",
                            },
                        ],
                        RetryAfterSeconds: 0,
                    },
                    Code: 404,
                },
            }
        occurred
    kueue_operator.go:100: Collecting Kueue diagnostics...
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration826826063
    kueue_operator.go:100: Stored Kueue CR state
    kueue_operator.go:100: Failed to get DSC for diagnostics: the server could not find the requested resource
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-fds8l/manager
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-tt77n/manager

=== FAIL: tests/trainer TestKueueWorkloadPreemptionSuspendsTrainJob (0.13s)
    kueue_operator.go:110: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster...
    kueue_operator.go:112: 
        Should be able to set DSC kueue to Unmanaged
        Unexpected error:
            <*errors.StatusError | 0xb0e4d980be0>: 
            the server could not find the requested resource
            {
                ErrStatus: {
                    TypeMeta: {Kind: "", APIVersion: ""},
                    ListMeta: {
                        SelfLink: "",
                        ResourceVersion: "",
                        Continue: "",
                        RemainingItemCount: nil,
                    },
                    Status: "Failure",
                    Message: "the server could not find the requested resource",
                    Reason: "NotFound",
                    Details: {
                        Name: "",
                        Group: "",
                        Kind: "",
                        UID: "",
                        Causes: [
                            {
                                Type: "UnexpectedServerResponse",
                                Message: "404 page not found",
                                Field: "",
                            },
                        ],
                        RetryAfterSeconds: 0,
                    },
                    Code: 404,
                },
            }
        occurred
    kueue_operator.go:100: Collecting Kueue diagnostics...
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueWorkloadPreemptionSuspendsTrainJob2549343212
    kueue_operator.go:100: Stored Kueue CR state
    kueue_operator.go:100: Failed to get DSC for diagnostics: the server could not find the requested resource
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-fds8l/manager
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-tt77n/manager

=== FAIL: tests/trainer TestKueueSchedulingReachesTrainJobPods (0.12s)
    kueue_operator.go:110: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster...
    kueue_operator.go:112: 
        Should be able to set DSC kueue to Unmanaged
        Unexpected error:
            <*errors.StatusError | 0xb0e4d6a0be0>: 
            the server could not find the requested resource
            {
                ErrStatus: {
                    TypeMeta: {Kind: "", APIVersion: ""},
                    ListMeta: {
                        SelfLink: "",
                        ResourceVersion: "",
                        Continue: "",
                        RemainingItemCount: nil,
                    },
                    Status: "Failure",
                    Message: "the server could not find the requested resource",
                    Reason: "NotFound",
                    Details: {
                        Name: "",
                        Group: "",
                        Kind: "",
                        UID: "",
                        Causes: [
                            {
                                Type: "UnexpectedServerResponse",
                                Message: "404 page not found",
                                Field: "",
                            },
                        ],
                        RetryAfterSeconds: 0,
                    },
                    Code: 404,
                },
            }
        occurred
    kueue_operator.go:100: Collecting Kueue diagnostics...
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueSchedulingReachesTrainJobPods1678157793
    kueue_operator.go:100: Stored Kueue CR state
    kueue_operator.go:100: Failed to get DSC for diagnostics: the server could not find the requested resource
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-fds8l/manager
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-9f8886599-tt77n/manager

DONE 67 tests, 61 skipped, 4 failures in 298.386s