go: downloading github.com/opendatahub-io/trainer/v2 v2.0.0-20261015092400-29a85e37fa9b
go: downloading sigs.k8s.io/kueue v0.15.7
go: downloading github.com/kubeflow/training-operator v1.7.0
go: downloading github.com/openshift/api v0.0.0-20260805215214-cfb63858e9d7
go: downloading github.com/openshift/client-go v0.0.0-20251015124057-db0dee36e235
go: downloading github.com/openshift/kueue-operator v0.0.0-20251202204851-958c48004dad
go: downloading github.com/operator-framework/api v0.36.0
go: downloading github.com/operator-framework/operator-lifecycle-manager v0.38.0
go: downloading github.com/ray-project/kuberay/ray-operator v1.5.1
go: downloading github.com/minio/minio-go/v7 v7.0.98
go: downloading github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674
go: downloading k8s.io/streaming v0.37.1
go: downloading github.com/sirupsen/logrus v1.9.3
go: downloading github.com/golang-jwt/jwt/v5 v5.3.1
go: downloading github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
go: downloading golang.org/x/term v0.46.0
go: downloading github.com/google/gnostic-models v0.7.1
go: downloading github.com/moby/spdystream v0.5.1
go: downloading golang.org/x/text v0.42.0
go: downloading golang.org/x/sys v0.48.0
go: downloading github.com/jpillora/backoff v1.0.0
go: downloading github.com/blang/semver/v4 v4.0.0
go: downloading github.com/dustin/go-humanize v1.0.1
go: downloading github.com/klauspost/compress v1.20.1
go: downloading github.com/klauspost/crc32 v1.3.0
go: downloading github.com/minio/crc64nvme v1.1.1
go: downloading github.com/minio/md5-simd v1.1.2
go: downloading github.com/go-ini/ini v1.67.0
go: downloading github.com/klauspost/cpuid/v2 v2.2.11
go: downloading golang.org/x/crypto v0.57.0
go: downloading github.com/rs/xid v1.6.0
go: downloading github.com/tinylib/msgp v1.6.1
go: downloading github.com/philhofer/fwd v1.2.0
Initial Kueue managementState: 
=== RUN   TestOpenMPICudaClusterTrainingRuntime
    utils_runtimes.go:183: Using image from ClusterTrainingRuntime "openmpi-cuda": quay.io/opendatahub/odh-th-torch-cuda-py312:odh-stable
    cluster_training_runtimes_test.go:185: ClusterTrainingRuntime 'openmpi-cuda' is present, uses image quay.io/opendatahub/odh-th-torch-cuda-py312:odh-stable, and is labeled for framework openmpi
--- PASS: TestOpenMPICudaClusterTrainingRuntime (0.03s)
=== RUN   TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes
    cluster_training_runtimes_test.go:219: CTR "training-hub" matches DefaultClusterTrainingRuntime "torch-distributed"
    cluster_training_runtimes_test.go:219: CTR "training-hub-cpu" matches DefaultClusterTrainingRuntime "torch-distributed-cpu"
    cluster_training_runtimes_test.go:219: CTR "torch-distributed-cuda130-torch211-py312" matches DefaultClusterTrainingRuntime "torch-distributed"
    cluster_training_runtimes_test.go:219: CTR "torch-distributed-rocm714-torch211-py312" matches DefaultClusterTrainingRuntime "torch-distributed-rocm"
    cluster_training_runtimes_test.go:219: CTR "training-hub-rocm" matches DefaultClusterTrainingRuntime "torch-distributed-rocm"
    cluster_training_runtimes_test.go:219: CTR "training-hub-th09-cuda130-torch211-py312" matches DefaultClusterTrainingRuntime "torch-distributed"
    cluster_training_runtimes_test.go:219: CTR "training-hub-th09-cpu-torch211-py312" matches DefaultClusterTrainingRuntime "torch-distributed-cpu"
    cluster_training_runtimes_test.go:219: CTR "training-hub-th09-rocm714-torch211-py312" matches DefaultClusterTrainingRuntime "torch-distributed-rocm"
    cluster_training_runtimes_test.go:219: CTR "torch-distributed-cpu-torch211-py312" matches DefaultClusterTrainingRuntime "torch-distributed-cpu"
    cluster_training_runtimes_test.go:222: All CTRs match their DefaultClusterTrainingRuntime counterparts!
--- PASS: TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes (0.03s)
=== RUN   TestRunTrainJobWithDefaultClusterTrainingRuntimes
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'
--- SKIP: TestRunTrainJobWithDefaultClusterTrainingRuntimes (0.00s)
=== RUN   TestJobSetWorkflow
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'
--- SKIP: TestJobSetWorkflow (0.00s)
=== RUN   TestFailedJobSetWorkflow
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestFailedJobSetWorkflow (0.00s)
=== RUN   TestKubeflowSdk
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'
--- SKIP: TestKubeflowSdk (0.00s)
=== RUN   TestKubeflowSdkKueueIntegration
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'
--- SKIP: TestKubeflowSdkKueueIntegration (0.00s)
=== RUN   TestKubeflowSdkOpenMPICudaKueueIntegration
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestKubeflowSdkOpenMPICudaKueueIntegration (0.00s)
=== RUN   TestSftTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestSftTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestOsftTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestOsftTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestLoraTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestLoraTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestGrpoTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestGrpoTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestSftCallbacksTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestSftCallbacksTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestLoraCallbacksTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestLoraCallbacksTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestOsftCallbacksTrainingHubSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestOsftCallbacksTrainingHubSingleNodeSingleGPU (0.00s)
=== RUN   TestOsftTrainingHubMultiNodeMultiGPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestOsftTrainingHubMultiNodeMultiGPU (0.00s)
=== RUN   TestLoraTrainingHubMultiNodeMultiGPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestLoraTrainingHubMultiNodeMultiGPU (0.00s)
=== RUN   TestSftTrainingHubMultiNodeMultiGPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestSftTrainingHubMultiNodeMultiGPU (0.00s)
=== RUN   TestRhaiTrainingProgressionCPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestRhaiTrainingProgressionCPU (0.00s)
=== RUN   TestRhaiJitCheckpointingCPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestRhaiJitCheckpointingCPU (0.00s)
=== RUN   TestRhaiFeaturesCPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestRhaiFeaturesCPU (0.00s)
=== RUN   TestRhaiTrainingProgressionCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiTrainingProgressionCuda (0.00s)
=== RUN   TestRhaiJitCheckpointingCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiJitCheckpointingCuda (0.00s)
=== RUN   TestRhaiFeaturesCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiFeaturesCuda (0.00s)
=== RUN   TestRhaiTrainingProgressionRocm
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiTrainingProgressionRocm (0.00s)
=== RUN   TestRhaiJitCheckpointingRocm
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiJitCheckpointingRocm (0.00s)
=== RUN   TestRhaiFeaturesRocm
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiFeaturesRocm (0.00s)
=== RUN   TestRhaiTrainingProgressionMultiGpuCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiTrainingProgressionMultiGpuCuda (0.00s)
=== RUN   TestRhaiJitCheckpointingMultiGpuCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiJitCheckpointingMultiGpuCuda (0.00s)
=== RUN   TestRhaiFeaturesMultiGpuCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiFeaturesMultiGpuCuda (0.00s)
=== RUN   TestRhaiTrainingProgressionMultiGpuRocm
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiTrainingProgressionMultiGpuRocm (0.00s)
=== RUN   TestRhaiJitCheckpointingMultiGpuRocm
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiJitCheckpointingMultiGpuRocm (0.00s)
=== RUN   TestRhaiFeaturesMultiGpuRocm
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestRhaiFeaturesMultiGpuRocm (0.00s)
=== RUN   TestTrainingFailureScenarios
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestTrainingFailureScenarios (0.00s)
=== RUN   TestTorchrunTrainingFailure
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestTorchrunTrainingFailure (0.00s)
=== RUN   TestRhaiS3CheckpointingCPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestRhaiS3CheckpointingCPU (0.00s)
=== RUN   TestRhaiS3FsdpFullStateCheckpointingCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3FsdpFullStateCheckpointingCuda (0.00s)
=== RUN   TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda (0.00s)
=== RUN   TestRhaiS3FsdpSharedStateCheckpointingCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3FsdpSharedStateCheckpointingCuda (0.00s)
=== RUN   TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda (0.00s)
=== RUN   TestRhaiS3DeepspeedStage0CheckpointingCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3DeepspeedStage0CheckpointingCuda (0.00s)
=== RUN   TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda (0.00s)
=== RUN   TestRhaiSpeculatorPipelineSingleGpuCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiSpeculatorPipelineSingleGpuCuda (0.00s)
=== RUN   TestRhaiSpeculatorPipelineMultiGpuCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiSpeculatorPipelineMultiGpuCuda (0.00s)
=== RUN   TestRhaiSpeculatorOfflineSingleGpuCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiSpeculatorOfflineSingleGpuCuda (0.00s)
=== RUN   TestRhaiSpeculatorOfflineMultiGpuCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiSpeculatorOfflineMultiGpuCuda (0.00s)
=== RUN   TestRhaiSpeculatorOnlineSingleGpuCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiSpeculatorOnlineSingleGpuCuda (0.00s)
=== RUN   TestRhaiSpeculatorOnlineMultiGpuCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestRhaiSpeculatorOnlineMultiGpuCuda (0.00s)
=== RUN   TestDraFieldsOnTrainJobReachPods
I1023 03:59:53.451329   18736 warnings.go:107] "Warning: spec.trainer.numProcPerNode is not set and the node container uses DRA claims; the GPU count is not derived from the ResourceClaimTemplate, so torchrun will use the CPU count. Set numProcPerNode explicitly"
    test.go:171: Retrieving Pod Container test-ns-99dnv/test-dra-trainjob-6ssxn-node-0-0-k89gf/node logs
    test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir
    test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestDraFieldsOnTrainJobReachPods1695036070
    test.go:171: Retrieving Pod Container test-ns-99dnv/test-dra-trainjob-6ssxn-node-0-1-47f8v/node logs
--- PASS: TestDraFieldsOnTrainJobReachPods (1.27s)
=== RUN   TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 (0.00s)
=== RUN   TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda (0.00s)
=== RUN   TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda (0.00s)
=== RUN   TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda (0.00s)
=== RUN   TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda (0.00s)
=== RUN   TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm (0.00s)
=== RUN   TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm (0.00s)
=== RUN   TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm (0.00s)
=== RUN   TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm (0.00s)
=== RUN   TestKueueWorkloadPreemptionSuspendsTrainJob
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'
--- SKIP: TestKueueWorkloadPreemptionSuspendsTrainJob (0.00s)
=== RUN   TestKueueWorkloadInadmissibleWithNonExistentLocalQueue
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestKueueWorkloadInadmissibleWithNonExistentLocalQueue (0.00s)
=== RUN   TestKueueSchedulingReachesTrainJobPods
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'
--- SKIP: TestKueueSchedulingReachesTrainJobPods (0.00s)
=== RUN   TestSetupUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:83: Skip due to issue RHOAIENG-48867
--- SKIP: TestSetupUpgradeTrainJob (0.00s)
=== RUN   TestRunUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:158: Skip due to issue RHOAIENG-48867
--- SKIP: TestRunUpgradeTrainJob (0.00s)
=== RUN   TestSetupSpecificRuntimeUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:205: Skip due to issue RHOAIENG-48867
--- SKIP: TestSetupSpecificRuntimeUpgradeTrainJob (0.00s)
=== RUN   TestRunSpecificRuntimeUpgradeTrainJob
    trainer_kueue_upgrade_training_test.go:295: Skip due to issue RHOAIENG-48867
--- SKIP: TestRunSpecificRuntimeUpgradeTrainJob (0.00s)
=== RUN   TestSetupCustomRuntimeUpgradeTrainJob
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Pre-Upgrade'
--- SKIP: TestSetupCustomRuntimeUpgradeTrainJob (0.00s)
=== RUN   TestRunCustomRuntimeUpgradeTrainJob
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Post-Upgrade'
--- SKIP: TestRunCustomRuntimeUpgradeTrainJob (0.00s)
=== RUN   TestTrainerSecureServing
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerSecureServing (0.00s)
=== RUN   TestTrainerPrometheusScrape
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerPrometheusScrape (0.00s)
=== RUN   TestMultiNodeOpenMPICpuTrainJob
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'
--- SKIP: TestMultiNodeOpenMPICpuTrainJob (0.00s)
=== RUN   TestMultiNodeOpenMPITrainJob
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestMultiNodeOpenMPITrainJob (0.00s)
=== RUN   TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics (0.00s)
=== RUN   TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth (0.00s)
=== RUN   TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer (0.00s)
=== RUN   TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort (0.00s)
=== RUN   TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort (0.00s)
=== RUN   TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'
--- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort (0.00s)
=== RUN   TestOpenMPICudaTrainJobKueueIntegration
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestOpenMPICudaTrainJobKueueIntegration (0.00s)
=== RUN   TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate (0.00s)
=== RUN   TestSftStockTrlSingleNodeSingleGPU
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'
--- SKIP: TestSftStockTrlSingleNodeSingleGPU (0.00s)
=== RUN   TestSftStockTrlSingleNodeSingleGPUWithTorchRocm
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'
--- SKIP: TestSftStockTrlSingleNodeSingleGPUWithTorchRocm (0.00s)
=== RUN   TestTrainerTLSProfileWatcher
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier3'
--- SKIP: TestTrainerTLSProfileWatcher (0.00s)
=== RUN   TestSetupTrainingRuntime
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Pre-Upgrade'
--- SKIP: TestSetupTrainingRuntime (0.00s)
=== RUN   TestVerifyTrainingRuntime
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Post-Upgrade'
--- SKIP: TestVerifyTrainingRuntime (0.00s)
=== RUN   TestSetupSleepTrainJob
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Pre-Upgrade'
--- SKIP: TestSetupSleepTrainJob (0.00s)
=== RUN   TestVerifySleepTrainJob
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Post-Upgrade'
--- SKIP: TestVerifySleepTrainJob (0.00s)
PASS
TearDown: Skipping Kueue teardown because no DataScienceCluster is installed
ok  	github.com/opendatahub-io/distributed-workloads/tests/trainer	1.415s

=== Skipped
=== SKIP: tests/trainer TestRunTrainJobWithDefaultClusterTrainingRuntimes (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'

=== SKIP: tests/trainer TestJobSetWorkflow (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'

=== SKIP: tests/trainer TestFailedJobSetWorkflow (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestKubeflowSdk (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'

=== SKIP: tests/trainer TestKubeflowSdkKueueIntegration (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'

=== SKIP: tests/trainer TestKubeflowSdkOpenMPICudaKueueIntegration (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestSftTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestOsftTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestLoraTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestGrpoTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestSftCallbacksTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestLoraCallbacksTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestOsftCallbacksTrainingHubSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestOsftTrainingHubMultiNodeMultiGPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestLoraTrainingHubMultiNodeMultiGPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestSftTrainingHubMultiNodeMultiGPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiTrainingProgressionCPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestRhaiJitCheckpointingCPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestRhaiFeaturesCPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestRhaiTrainingProgressionCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiJitCheckpointingCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiFeaturesCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiTrainingProgressionRocm (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiJitCheckpointingRocm (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiFeaturesRocm (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiTrainingProgressionMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiJitCheckpointingMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiFeaturesMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiTrainingProgressionMultiGpuRocm (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiJitCheckpointingMultiGpuRocm (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestRhaiFeaturesMultiGpuRocm (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestTrainingFailureScenarios (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestTorchrunTrainingFailure (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3CheckpointingCPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestRhaiS3FsdpFullStateCheckpointingCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3FsdpSharedStateCheckpointingCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3DeepspeedStage0CheckpointingCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiSpeculatorPipelineSingleGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiSpeculatorPipelineMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiSpeculatorOfflineSingleGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiSpeculatorOfflineMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiSpeculatorOnlineSingleGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestRhaiSpeculatorOnlineMultiGpuCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestKueueWorkloadPreemptionSuspendsTrainJob (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'

=== SKIP: tests/trainer TestKueueWorkloadInadmissibleWithNonExistentLocalQueue (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestKueueSchedulingReachesTrainJobPods (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'

=== SKIP: tests/trainer TestSetupUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:83: Skip due to issue RHOAIENG-48867

=== SKIP: tests/trainer TestRunUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:158: Skip due to issue RHOAIENG-48867

=== SKIP: tests/trainer TestSetupSpecificRuntimeUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:205: Skip due to issue RHOAIENG-48867

=== SKIP: tests/trainer TestRunSpecificRuntimeUpgradeTrainJob (0.00s)
    trainer_kueue_upgrade_training_test.go:295: Skip due to issue RHOAIENG-48867

=== SKIP: tests/trainer TestSetupCustomRuntimeUpgradeTrainJob (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Pre-Upgrade'

=== SKIP: tests/trainer TestRunCustomRuntimeUpgradeTrainJob (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Post-Upgrade'

=== SKIP: tests/trainer TestTrainerSecureServing (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestTrainerPrometheusScrape (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestMultiNodeOpenMPICpuTrainJob (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier1'

=== SKIP: tests/trainer TestMultiNodeOpenMPITrainJob (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier2'

=== SKIP: tests/trainer TestOpenMPICudaTrainJobKueueIntegration (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestSftStockTrlSingleNodeSingleGPU (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-CUDA'

=== SKIP: tests/trainer TestSftStockTrlSingleNodeSingleGPUWithTorchRocm (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'KFTO-ROCm'

=== SKIP: tests/trainer TestTrainerTLSProfileWatcher (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Tier3'

=== SKIP: tests/trainer TestSetupTrainingRuntime (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Pre-Upgrade'

=== SKIP: tests/trainer TestVerifyTrainingRuntime (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Post-Upgrade'

=== SKIP: tests/trainer TestSetupSleepTrainJob (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Pre-Upgrade'

=== SKIP: tests/trainer TestVerifySleepTrainJob (0.00s)
    test_tag.go:37: Test tier 'Smoke' doesn't match expected tier 'Post-Upgrade'

DONE 86 tests, 83 skipped in 1.416s