Initial Kueue managementState: 
=== RUN   TestDefaultClusterTrainingRuntimes
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'
--- SKIP: TestDefaultClusterTrainingRuntimes (0.00s)
=== RUN   TestOpenMPICudaClusterTrainingRuntime
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'
--- SKIP: TestOpenMPICudaClusterTrainingRuntime (0.00s)
=== RUN   TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'
--- SKIP: TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes (0.00s)
=== RUN   TestRunTrainJobWithDefaultClusterTrainingRuntimes
    cluster_training_runtimes_test.go:258: Running TrainJob with ClusterTrainingRuntime: torch-distributed
    cluster_training_runtimes_test.go:264: Created TrainJob test-ns-fvvng/test-trainjob-qjjrx successfully
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestRunTrainJobWithDefaultClusterTrainingRuntimes2586800953
    cluster_training_runtimes_test.go:275: TrainJob with ClusterTrainingRuntime 'torch-distributed' completed successfully
    cluster_training_runtimes_test.go:277: No ODH/RHOAI operator subscription found; assuming standalone installation
    cluster_training_runtimes_test.go:258: Running TrainJob with ClusterTrainingRuntime: torch-distributed-rocm
    cluster_training_runtimes_test.go:264: Created TrainJob test-ns-k4f86/test-trainjob-9hfpx successfully
    cluster_training_runtimes_test.go:275: TrainJob with ClusterTrainingRuntime 'torch-distributed-rocm' completed successfully
    cluster_training_runtimes_test.go:277: No ODH/RHOAI operator subscription found; assuming standalone installation
    cluster_training_runtimes_test.go:258: Running TrainJob with ClusterTrainingRuntime: torch-distributed-cpu
    cluster_training_runtimes_test.go:264: Created TrainJob test-ns-z92lw/test-trainjob-wnrwq successfully
    cluster_training_runtimes_test.go:275: TrainJob with ClusterTrainingRuntime 'torch-distributed-cpu' completed successfully
    cluster_training_runtimes_test.go:277: No ODH/RHOAI operator subscription found; assuming standalone installation
    cluster_training_runtimes_test.go:245: Skipping MPI runtime 'openmpi-cuda' (covered by TestMultiNodeOpenMPITrainJob)
    cluster_training_runtimes_test.go:245: Skipping MPI runtime 'openmpi-cpu' (covered by TestMultiNodeOpenMPITrainJob)
    cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'torch-distributed-cuda130-torch211-py312' (image 'odh-th-torch-cuda-py312' already tested)
    cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'torch-distributed-rocm714-torch211-py312' (image 'odh-th-torch-rocm-py312' already tested)
    cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'torch-distributed-cpu-torch211-py312' (image 'odh-th-torch-cpu-py312' already tested)
    cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'training-hub' (image 'odh-th-torch-cuda-py312' already tested)
    cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'training-hub-cpu' (image 'odh-th-torch-cpu-py312' already tested)
    cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'training-hub-rocm' (image 'odh-th-torch-rocm-py312' already tested)
    cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'training-hub-th09-cuda130-torch211-py312' (image 'odh-th-torch-cuda-py312' already tested)
    cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'training-hub-th09-cpu-torch211-py312' (image 'odh-th-torch-cpu-py312' already tested)
    cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'training-hub-th09-rocm714-torch211-py312' (image 'odh-th-torch-rocm-py312' already tested)
    cluster_training_runtimes_test.go:249: Skipping speculator runtime 'vllm-extract-cuda' (covered by speculator tests)
    cluster_training_runtimes_test.go:249: Skipping speculator runtime 'speculator-model-opt-cuda' (covered by speculator tests)
    cluster_training_runtimes_test.go:290: All TrainJobs with expected ClusterTrainingRuntimes completed successfully !!!
    test.go:171: Retrieving Pod Container test-ns-z92lw/test-trainjob-wnrwq-node-0-0-p7zp6/node logs
    test.go:171: Retrieving Pod Container test-ns-k4f86/test-trainjob-9hfpx-node-0-0-jt264/node logs
    test.go:171: Retrieving Pod Container test-ns-fvvng/test-trainjob-qjjrx-node-0-0-xjgqs/node logs
--- PASS: TestRunTrainJobWithDefaultClusterTrainingRuntimes (993.28s)
=== RUN   TestJobSetWorkflow
    jobset_workflow_test.go:45: Created PersistentVolumeClaim test-ns-xprrk/pvc-rbqrt successfully
    utils_runtimes.go:183: Using image from ClusterTrainingRuntime "torch-distributed-cpu": quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1140
    jobset_workflow_test.go:52: Created TrainingRuntime test-ns-xprrk/test-trainingruntime-xdxnb
    jobset_workflow_test.go:55: Created TrainJob test-ns-xprrk/test-trainjob-5z6vq
    jobset_workflow_test.go:61: JobSet created with 3 replicated jobs (dataset-initializer, model-initializer, node)
    jobset_workflow_test.go:64: Monitoring job execution ...
    jobset_workflow_test.go:64: dataset-initializer job is created: test-trainjob-5z6vq-dataset-initializer-0
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestJobSetWorkflow1167697452
    jobset_workflow_test.go:64: dataset-initializer job is completed: test-trainjob-5z6vq-dataset-initializer-0
    jobset_workflow_test.go:64: model-initializer job is created: test-trainjob-5z6vq-model-initializer-0
    jobset_workflow_test.go:64: model-initializer job is completed: test-trainjob-5z6vq-model-initializer-0
    jobset_workflow_test.go:64: node job is created: test-trainjob-5z6vq-node-0
    jobset_workflow_test.go:64: Sequential job execution is verified successfully: dataset-initializer → model-initializer → node
    jobset_workflow_test.go:69: TrainJob test-ns-xprrk/test-trainjob-5z6vq completed
    test.go:171: Retrieving Pod Container test-ns-xprrk/test-trainjob-5z6vq-dataset-initializer-0-0-gp6dt/dataset-initializer logs
    test.go:171: Retrieving Pod Container test-ns-xprrk/test-trainjob-5z6vq-model-initializer-0-0-cdx6f/model-initializer logs
    test.go:171: Retrieving Pod Container test-ns-xprrk/test-trainjob-5z6vq-node-0-0-khg98/node logs
--- PASS: TestJobSetWorkflow (153.79s)
=== RUN   TestFailedJobSetWorkflow
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestFailedJobSetWorkflow (0.00s)
=== RUN   TestKubeflowSdk
    fashion_mnist_tests.go:56: Created User RoleBinding test-ns-tdnz6/rb-xnlvv successfully
    fashion_mnist_tests.go:57: Created User RoleBinding test-ns-tdnz6/rb-7rdkg successfully
    fashion_mnist_tests.go:57: Created ClusterRole /clusterrole-chpc2 successfully
    fashion_mnist_tests.go:57: Created User ClusterRoleBinding crb-b5trf successfully
    fashion_mnist_tests.go:64: Created ConfigMap test-ns-tdnz6/config-8ngq6 successfully
    fashion_mnist_tests.go:82: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX.
    fashion_mnist_tests.go:84: Created PersistentVolumeClaim test-ns-tdnz6/pvc-n7pzh successfully
    fashion_mnist_tests.go:93: OpenShift API URL: https://a97f8398d4dc944878cfd92eb7b75d4e-7a4ecc4a4c388e7e.elb.us-east-2.amazonaws.com:6443
    dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications
    environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1140
    notebook.go:44: Created test Deployment test-ns-tdnz6/dw-deployment-f8sq7
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdk1481802450
    fashion_mnist_tests.go:138: 
        Deployment runner execution reported FAILURE
        Unexpected error:
            <*errors.errorString | 0x2fdce6f4f3d0>: 
            pod execution failed
            {
                s: "pod execution failed",
            }
        occurred
    test.go:171: Retrieving Pod Container test-ns-tdnz6/dw-deployment-f8sq7-797844fc4b-7mt2w/notebook logs
--- FAIL: TestKubeflowSdk (13.70s)
=== RUN   TestKubeflowSdkKueueIntegration
    kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK
    kueue_operator.go:227: Kueue CR 'cluster' exists
    kueue_operator.go:229: Waiting for Kueue CR to be ready...
    kueue_operator.go:236: Kueue CR is ready
    kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR...
    kueue_operator.go:243: TrainJob framework is present in Kueue CR
    kueue_operator.go:239: Verifying Deployment framework is present in Kueue CR...
    kueue_operator.go:243: Deployment framework is present in Kueue CR
    fashion_mnist_tests.go:148: Created Kueue-managed namespace: test-ns-jxhrf
    fashion_mnist_tests.go:153: Created User RoleBinding test-ns-jxhrf/rb-7vvlx successfully
    fashion_mnist_tests.go:154: Created User RoleBinding test-ns-jxhrf/rb-2vjbp successfully
    fashion_mnist_tests.go:154: Created ClusterRole /clusterrole-h4lq6 successfully
    fashion_mnist_tests.go:154: Created User ClusterRoleBinding crb-jw2pt successfully
    fashion_mnist_tests.go:157: Created Kueue ResourceFlavor rf-5kg2d successfully
    fashion_mnist_tests.go:184: Created Kueue ClusterQueue cq-ssz5w successfully
    fashion_mnist_tests.go:189: Created Kueue LocalQueue test-ns-jxhrf/lq-zkkb9 successfully
    fashion_mnist_tests.go:190: Created custom LocalQueue lq-zkkb9 for TrainJob
    fashion_mnist_tests.go:197: Created ConfigMap test-ns-jxhrf/config-8mpd4 successfully
    fashion_mnist_tests.go:215: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX.
    fashion_mnist_tests.go:217: Created PersistentVolumeClaim test-ns-jxhrf/pvc-xgpvx successfully
    fashion_mnist_tests.go:226: OpenShift API URL: https://a97f8398d4dc944878cfd92eb7b75d4e-7a4ecc4a4c388e7e.elb.us-east-2.amazonaws.com:6443
    dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications
    environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1140
    notebook.go:44: Created test Deployment test-ns-jxhrf/dw-deployment-hcnpm
    fashion_mnist_tests.go:271: Verifying SDK-submitted TrainJob has custom queue label: lq-zkkb9
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration3181399757
    fashion_mnist_tests.go:272: 
        Timed out after 1200.001s.
        Expected
            <[]v1alpha1.TrainJob | len:0, cap:0>: []
        to have length 1
    test.go:171: Retrieving Pod Container test-ns-jxhrf/dw-deployment-hcnpm-58d569f9dd-r225h/notebook logs
    kueue_operator.go:100: Collecting Kueue diagnostics...
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration1541199642
    kueue_operator.go:100: Stored Kueue CR state
    kueue_operator.go:100: DataScienceCluster is not installed; skipping DSC diagnostics
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-6f9bf978cc-l6rsc/manager
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-6f9bf978cc-z8b62/manager
--- FAIL: TestKubeflowSdkKueueIntegration (1200.97s)
=== RUN   TestKubeflowSdkOpenMPICudaKueueIntegration
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestKubeflowSdkOpenMPICudaKueueIntegration (0.00s)
=== RUN   TestSftTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestSftTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestOsftTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestOsftTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestLoraTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestLoraTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestGrpoTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestGrpoTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestSftCallbacksTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestSftCallbacksTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestLoraCallbacksTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestLoraCallbacksTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestOsftCallbacksTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestOsftCallbacksTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestOsftTrainingHubMultiNodeMultiGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestOsftTrainingHubMultiNodeMultiGPU (0.00s)
=== RUN   TestLoraTrainingHubMultiNodeMultiGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestLoraTrainingHubMultiNodeMultiGPU (0.00s)
=== RUN   TestSftTrainingHubMultiNodeMultiGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestSftTrainingHubMultiNodeMultiGPU (0.00s)
=== RUN   TestRhaiTrainingProgressionCPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestRhaiTrainingProgressionCPU (0.00s)
=== RUN   TestRhaiJitCheckpointingCPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestRhaiJitCheckpointingCPU (0.00s)
=== RUN   TestRhaiFeaturesCPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestRhaiFeaturesCPU (0.00s)
=== RUN   TestRhaiTrainingProgressionCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiTrainingProgressionCuda (0.00s)
=== RUN   TestRhaiJitCheckpointingCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiJitCheckpointingCuda (0.00s)
=== RUN   TestRhaiFeaturesCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiFeaturesCuda (0.00s)
=== RUN   TestRhaiTrainingProgressionRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiTrainingProgressionRocm (0.00s)
=== RUN   TestRhaiJitCheckpointingRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiJitCheckpointingRocm (0.00s)
=== RUN   TestRhaiFeaturesRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiFeaturesRocm (0.00s)
=== RUN   TestRhaiTrainingProgressionMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiTrainingProgressionMultiGpuCuda (0.00s)
=== RUN   TestRhaiJitCheckpointingMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiJitCheckpointingMultiGpuCuda (0.00s)
=== RUN   TestRhaiFeaturesMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiFeaturesMultiGpuCuda (0.00s)
=== RUN   TestRhaiTrainingProgressionMultiGpuRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiTrainingProgressionMultiGpuRocm (0.00s)
=== RUN   TestRhaiJitCheckpointingMultiGpuRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiJitCheckpointingMultiGpuRocm (0.00s)
=== RUN   TestRhaiFeaturesMultiGpuRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiFeaturesMultiGpuRocm (0.00s)
=== RUN   TestTrainingFailureScenarios
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestTrainingFailureScenarios (0.00s)
=== RUN   TestTorchrunTrainingFailure
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestTorchrunTrainingFailure (0.00s)
=== RUN   TestRhaiS3CheckpointingCPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestRhaiS3CheckpointingCPU (0.00s)
=== RUN   TestRhaiS3FsdpFullStateCheckpointingCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3FsdpFullStateCheckpointingCuda (0.00s)
=== RUN   TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda (0.00s)
=== RUN   TestRhaiS3FsdpSharedStateCheckpointingCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3FsdpSharedStateCheckpointingCuda (0.00s)
=== RUN   TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda (0.00s)
=== RUN   TestRhaiS3DeepspeedStage0CheckpointingCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3DeepspeedStage0CheckpointingCuda (0.00s)
=== RUN   TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda (0.00s)
=== RUN   TestRhaiSpeculatorPipelineSingleGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiSpeculatorPipelineSingleGpuCuda (0.00s)
=== RUN   TestRhaiSpeculatorPipelineMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiSpeculatorPipelineMultiGpuCuda (0.00s)
=== RUN   TestRhaiSpeculatorOfflineSingleGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiSpeculatorOfflineSingleGpuCuda (0.00s)
=== RUN   TestRhaiSpeculatorOfflineMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiSpeculatorOfflineMultiGpuCuda (0.00s)
=== RUN   TestRhaiSpeculatorOnlineSingleGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiSpeculatorOnlineSingleGpuCuda (0.00s)
=== RUN   TestRhaiSpeculatorOnlineMultiGpuCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiSpeculatorOnlineMultiGpuCuda (0.00s)
=== RUN   TestDraFieldsOnTrainJobReachPods
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'
--- SKIP: TestDraFieldsOnTrainJobReachPods (0.00s)
=== RUN   TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 (0.00s)
=== RUN   TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda (0.00s)
=== RUN   TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda (0.00s)
=== RUN   TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda (0.00s)
=== RUN   TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda (0.00s)
=== RUN   TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm (0.00s)
=== RUN   TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm (0.00s)
=== RUN   TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm (0.00s)
=== RUN   TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm (0.00s)
=== RUN   TestKueueWorkloadPreemptionSuspendsTrainJob
    kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK
    kueue_operator.go:227: Kueue CR 'cluster' exists
    kueue_operator.go:229: Waiting for Kueue CR to be ready...
    kueue_operator.go:236: Kueue CR is ready
    kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR...
    kueue_operator.go:243: TrainJob framework is present in Kueue CR
    trainer_kueue_integration_test.go:67: Created Kueue-managed namespace: test-ns-l5xzb
    trainer_kueue_integration_test.go:70: Created Kueue ResourceFlavor rf-c5znq successfully
    trainer_kueue_integration_test.go:72: Created Kueue ClusterQueue cq-xctlm successfully
    trainer_kueue_integration_test.go:96: Created Kueue LocalQueue test-ns-l5xzb/lq-qtcks successfully
    trainer_kueue_integration_test.go:123: Created TrainJob: test-preemption-trainjob-4nghp
    trainer_kueue_integration_test.go:133: Workload 'trainjob-test-preemption-trainjob-4nghp-8dfd1' is admitted
    trainer_kueue_integration_test.go:137: Verifying TrainJob pods are running...
    trainer_kueue_integration_test.go:147: TrainJob 'test-preemption-trainjob-4nghp' pods are running
    trainer_kueue_integration_test.go:150: User is preempting workload 'trainjob-test-preemption-trainjob-4nghp-8dfd1' now ...
    trainer_kueue_integration_test.go:158: Workload 'trainjob-test-preemption-trainjob-4nghp-8dfd1' is preempted
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueWorkloadPreemptionSuspendsTrainJob4002784705
    trainer_kueue_integration_test.go:164: Workload 'trainjob-test-preemption-trainjob-4nghp-8dfd1' is evicted
    trainer_kueue_integration_test.go:170: TrainJob 'test-preemption-trainjob-4nghp' is suspended
    trainer_kueue_integration_test.go:172: Workload preemption successfully suspended TrainJob 'test-preemption-trainjob-4nghp'!
    test.go:171: Retrieving Pod Container test-ns-l5xzb/test-preemption-trainjob-4nghp-node-0-0-zcxx7/node logs
--- PASS: TestKueueWorkloadPreemptionSuspendsTrainJob (3.63s)
=== RUN   TestKueueWorkloadInadmissibleWithNonExistentLocalQueue
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestKueueWorkloadInadmissibleWithNonExistentLocalQueue (0.00s)
=== RUN   TestKueueSchedulingReachesTrainJobPods
    kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK
    kueue_operator.go:227: Kueue CR 'cluster' exists
    kueue_operator.go:229: Waiting for Kueue CR to be ready...
    kueue_operator.go:236: Kueue CR is ready
    kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR...
    kueue_operator.go:243: TrainJob framework is present in Kueue CR
    trainer_kueue_integration_test.go:251: Created Kueue-managed namespace: test-ns-jj8qm
    trainer_kueue_integration_test.go:259: Created Kueue ResourceFlavor rf-sl25t successfully
    trainer_kueue_integration_test.go:274: Created Kueue ClusterQueue cq-kcmgp successfully
    trainer_kueue_integration_test.go:298: Created Kueue LocalQueue test-ns-jj8qm/lq-qbmpt successfully
    trainer_kueue_integration_test.go:332: Created TrainJob: test-runtimepatches-trainjob-jwr2h
    trainer_kueue_integration_test.go:340: Workload admitted
    trainer_kueue_integration_test.go:365: JobSet "test-runtimepatches-trainjob-jwr2h" pod templates carry the ResourceFlavor's scheduling directives
    test.go:171: Retrieving Pod Container test-ns-jj8qm/test-runtimepatches-trainjob-jwr2h-node-0-0-lctb5/node logs
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueSchedulingReachesTrainJobPods4005076377
--- PASS: TestKueueSchedulingReachesTrainJobPods (1.31s)
=== RUN   TestSetupUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:83: Skip due to issue RHOAIENG-48867
--- SKIP: TestSetupUpgradeTrainJob (0.00s)
=== RUN   TestRunUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:158: Skip due to issue RHOAIENG-48867
--- SKIP: TestRunUpgradeTrainJob (0.00s)
=== RUN   TestSetupSpecificRuntimeUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:205: Skip due to issue RHOAIENG-48867
--- SKIP: TestSetupSpecificRuntimeUpgradeTrainJob (0.00s)
=== RUN   TestRunSpecificRuntimeUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:295: Skip due to issue RHOAIENG-48867
--- SKIP: TestRunSpecificRuntimeUpgradeTrainJob (0.00s)
=== RUN   TestSetupCustomRuntimeUpgradeTrainJob
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade'
--- SKIP: TestSetupCustomRuntimeUpgradeTrainJob (0.00s)
=== RUN   TestRunCustomRuntimeUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:458: Skip due to issue RHOAIENG-87834
--- SKIP: TestRunCustomRuntimeUpgradeTrainJob (0.00s)
=== RUN   TestTrainerSecureServing
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerSecureServing (0.00s)
=== RUN   TestTrainerPrometheusScrape
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerPrometheusScrape (0.00s)
=== RUN   TestMultiNodeOpenMPICpuTrainJob
    trainer_mpi_test.go:51: Created ConfigMap test-ns-wdxh5/config-5hnbq successfully
    utils_runtimes.go:183: Using image from ClusterTrainingRuntime "openmpi-cpu": quay.io/opendatahub/odh-th-torch-cpu-py312:odh-stable
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestMultiNodeOpenMPICpuTrainJob2633662813
    test.go:171: Retrieving Pod Container test-ns-wdxh5/test-mpi-trainjob-p6x4t-launcher-0-0-fhmkp/node logs
    test.go:171: Retrieving Pod Container test-ns-wdxh5/test-mpi-trainjob-p6x4t-node-0-0-5vv6v/node logs
--- PASS: TestMultiNodeOpenMPICpuTrainJob (137.37s)
=== RUN   TestMultiNodeOpenMPITrainJob
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestMultiNodeOpenMPITrainJob (0.00s)
=== RUN   TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics (0.00s)
=== RUN   TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth (0.00s)
=== RUN   TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer (0.00s)
=== RUN   TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort (0.00s)
=== RUN   TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort (0.00s)
=== RUN   TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort (0.00s)
=== RUN   TestOpenMPICudaTrainJobKueueIntegration
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestOpenMPICudaTrainJobKueueIntegration (0.00s)
=== RUN   TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate (0.00s)
=== RUN   TestSftStockTrlSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestSftStockTrlSingleNodeSingleGPU (0.00s)
=== RUN   TestSftStockTrlSingleNodeSingleGPUWithTorchRocm
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestSftStockTrlSingleNodeSingleGPUWithTorchRocm (0.00s)
=== RUN   TestKubeflowTrainerSmoke
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'
--- SKIP: TestKubeflowTrainerSmoke (0.00s)
=== RUN   TestTrainerTLSProfileWatcher
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier3'
--- SKIP: TestTrainerTLSProfileWatcher (0.00s)
=== RUN   TestSetupTrainingRuntime
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade'
--- SKIP: TestSetupTrainingRuntime (0.00s)
=== RUN   TestVerifyTrainingRuntime
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade'
--- SKIP: TestVerifyTrainingRuntime (0.00s)
=== RUN   TestSetupSleepTrainJob
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade'
--- SKIP: TestSetupSleepTrainJob (0.00s)
=== RUN   TestVerifySleepTrainJob
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade'
--- SKIP: TestVerifySleepTrainJob (0.00s)
FAIL
TearDown: Skipping Kueue teardown because no DataScienceCluster is installed
FAIL	github.com/opendatahub-io/distributed-workloads/tests/trainer	2504.143s

=== Skipped
=== SKIP: tests/trainer TestDefaultClusterTrainingRuntimes (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'

=== SKIP: tests/trainer TestOpenMPICudaClusterTrainingRuntime (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'

=== SKIP: tests/trainer TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'

=== SKIP: tests/trainer TestFailedJobSetWorkflow (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestKubeflowSdkOpenMPICudaKueueIntegration (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestSftTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestOsftTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestLoraTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestGrpoTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestSftCallbacksTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestLoraCallbacksTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestOsftCallbacksTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestOsftTrainingHubMultiNodeMultiGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestLoraTrainingHubMultiNodeMultiGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestSftTrainingHubMultiNodeMultiGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiTrainingProgressionCPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestRhaiJitCheckpointingCPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestRhaiFeaturesCPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestRhaiTrainingProgressionCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiJitCheckpointingCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiFeaturesCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiTrainingProgressionRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiJitCheckpointingRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiFeaturesRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiTrainingProgressionMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiJitCheckpointingMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiFeaturesMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiTrainingProgressionMultiGpuRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiJitCheckpointingMultiGpuRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiFeaturesMultiGpuRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestTrainingFailureScenarios (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestTorchrunTrainingFailure (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3CheckpointingCPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestRhaiS3FsdpFullStateCheckpointingCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3FsdpSharedStateCheckpointingCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3DeepspeedStage0CheckpointingCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiSpeculatorPipelineSingleGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiSpeculatorPipelineMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiSpeculatorOfflineSingleGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiSpeculatorOfflineMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiSpeculatorOnlineSingleGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiSpeculatorOnlineMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestDraFieldsOnTrainJobReachPods (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestKueueWorkloadInadmissibleWithNonExistentLocalQueue (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestSetupUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:83: Skip due to issue RHOAIENG-48867

=== SKIP: tests/trainer TestRunUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:158: Skip due to issue RHOAIENG-48867

=== SKIP: tests/trainer TestSetupSpecificRuntimeUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:205: Skip due to issue RHOAIENG-48867

=== SKIP: tests/trainer TestRunSpecificRuntimeUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:295: Skip due to issue RHOAIENG-48867

=== SKIP: tests/trainer TestSetupCustomRuntimeUpgradeTrainJob (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade'

=== SKIP: tests/trainer TestRunCustomRuntimeUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:458: Skip due to issue RHOAIENG-87834

=== SKIP: tests/trainer TestTrainerSecureServing (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestTrainerPrometheusScrape (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestMultiNodeOpenMPITrainJob (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestOpenMPICudaTrainJobKueueIntegration (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestSftStockTrlSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestSftStockTrlSingleNodeSingleGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestKubeflowTrainerSmoke (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke'

=== SKIP: tests/trainer TestTrainerTLSProfileWatcher (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier3'

=== SKIP: tests/trainer TestSetupTrainingRuntime (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade'

=== SKIP: tests/trainer TestVerifyTrainingRuntime (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade'

=== SKIP: tests/trainer TestSetupSleepTrainJob (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade'

=== SKIP: tests/trainer TestVerifySleepTrainJob (0.00s)
    test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade'

=== Failed
=== FAIL: tests/trainer TestKubeflowSdk (13.70s)
    fashion_mnist_tests.go:56: Created User RoleBinding test-ns-tdnz6/rb-xnlvv successfully
    fashion_mnist_tests.go:57: Created User RoleBinding test-ns-tdnz6/rb-7rdkg successfully
    fashion_mnist_tests.go:57: Created ClusterRole /clusterrole-chpc2 successfully
    fashion_mnist_tests.go:57: Created User ClusterRoleBinding crb-b5trf successfully
    fashion_mnist_tests.go:64: Created ConfigMap test-ns-tdnz6/config-8ngq6 successfully
    fashion_mnist_tests.go:82: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX.
    fashion_mnist_tests.go:84: Created PersistentVolumeClaim test-ns-tdnz6/pvc-n7pzh successfully
    fashion_mnist_tests.go:93: OpenShift API URL: https://a97f8398d4dc944878cfd92eb7b75d4e-7a4ecc4a4c388e7e.elb.us-east-2.amazonaws.com:6443
    dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications
    environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1140
    notebook.go:44: Created test Deployment test-ns-tdnz6/dw-deployment-f8sq7
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdk1481802450
    fashion_mnist_tests.go:138: 
        Deployment runner execution reported FAILURE
        Unexpected error:
            <*errors.errorString | 0x2fdce6f4f3d0>: 
            pod execution failed
            {
                s: "pod execution failed",
            }
        occurred
    test.go:171: Retrieving Pod Container test-ns-tdnz6/dw-deployment-f8sq7-797844fc4b-7mt2w/notebook logs

=== FAIL: tests/trainer TestKubeflowSdkKueueIntegration (1200.97s)
    kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK
    kueue_operator.go:227: Kueue CR 'cluster' exists
    kueue_operator.go:229: Waiting for Kueue CR to be ready...
    kueue_operator.go:236: Kueue CR is ready
    kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR...
    kueue_operator.go:243: TrainJob framework is present in Kueue CR
    kueue_operator.go:239: Verifying Deployment framework is present in Kueue CR...
    kueue_operator.go:243: Deployment framework is present in Kueue CR
    fashion_mnist_tests.go:148: Created Kueue-managed namespace: test-ns-jxhrf
    fashion_mnist_tests.go:153: Created User RoleBinding test-ns-jxhrf/rb-7vvlx successfully
    fashion_mnist_tests.go:154: Created User RoleBinding test-ns-jxhrf/rb-2vjbp successfully
    fashion_mnist_tests.go:154: Created ClusterRole /clusterrole-h4lq6 successfully
    fashion_mnist_tests.go:154: Created User ClusterRoleBinding crb-jw2pt successfully
    fashion_mnist_tests.go:157: Created Kueue ResourceFlavor rf-5kg2d successfully
    fashion_mnist_tests.go:184: Created Kueue ClusterQueue cq-ssz5w successfully
    fashion_mnist_tests.go:189: Created Kueue LocalQueue test-ns-jxhrf/lq-zkkb9 successfully
    fashion_mnist_tests.go:190: Created custom LocalQueue lq-zkkb9 for TrainJob
    fashion_mnist_tests.go:197: Created ConfigMap test-ns-jxhrf/config-8mpd4 successfully
    fashion_mnist_tests.go:215: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX.
    fashion_mnist_tests.go:217: Created PersistentVolumeClaim test-ns-jxhrf/pvc-xgpvx successfully
    fashion_mnist_tests.go:226: OpenShift API URL: https://a97f8398d4dc944878cfd92eb7b75d4e-7a4ecc4a4c388e7e.elb.us-east-2.amazonaws.com:6443
    dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications
    environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1140
    notebook.go:44: Created test Deployment test-ns-jxhrf/dw-deployment-hcnpm
    fashion_mnist_tests.go:271: Verifying SDK-submitted TrainJob has custom queue label: lq-zkkb9
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration3181399757
    fashion_mnist_tests.go:272: 
        Timed out after 1200.001s.
        Expected
            <[]v1alpha1.TrainJob | len:0, cap:0>: []
        to have length 1
    test.go:171: Retrieving Pod Container test-ns-jxhrf/dw-deployment-hcnpm-58d569f9dd-r225h/notebook logs
    kueue_operator.go:100: Collecting Kueue diagnostics...
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration1541199642
    kueue_operator.go:100: Stored Kueue CR state
    kueue_operator.go:100: DataScienceCluster is not installed; skipping DSC diagnostics
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-6f9bf978cc-l6rsc/manager
    kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-6f9bf978cc-z8b62/manager

DONE 88 tests, 81 skipped, 2 failures in 2504.143s