Initial Kueue managementState: === RUN TestDefaultClusterTrainingRuntimes test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke' --- SKIP: TestDefaultClusterTrainingRuntimes (0.00s) === RUN TestOpenMPICudaClusterTrainingRuntime test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke' --- SKIP: TestOpenMPICudaClusterTrainingRuntime (0.00s) === RUN TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke' --- SKIP: TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes (0.00s) === RUN TestRunTrainJobWithDefaultClusterTrainingRuntimes cluster_training_runtimes_test.go:258: Running TrainJob with ClusterTrainingRuntime: torch-distributed cluster_training_runtimes_test.go:264: Created TrainJob test-ns-dm4m5/test-trainjob-26qsb successfully test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestRunTrainJobWithDefaultClusterTrainingRuntimes2044530762 cluster_training_runtimes_test.go:275: TrainJob with ClusterTrainingRuntime 'torch-distributed' completed successfully cluster_training_runtimes_test.go:277: No ODH/RHOAI operator subscription found; assuming standalone installation cluster_training_runtimes_test.go:258: Running TrainJob with ClusterTrainingRuntime: torch-distributed-rocm cluster_training_runtimes_test.go:264: Created TrainJob test-ns-d4qcn/test-trainjob-tcmjs successfully cluster_training_runtimes_test.go:275: TrainJob with ClusterTrainingRuntime 'torch-distributed-rocm' completed successfully cluster_training_runtimes_test.go:277: No ODH/RHOAI operator subscription found; assuming standalone installation cluster_training_runtimes_test.go:258: Running TrainJob with ClusterTrainingRuntime: torch-distributed-cpu cluster_training_runtimes_test.go:264: Created TrainJob test-ns-vl8sj/test-trainjob-4m5rt successfully cluster_training_runtimes_test.go:275: TrainJob with ClusterTrainingRuntime 'torch-distributed-cpu' completed successfully cluster_training_runtimes_test.go:277: No ODH/RHOAI operator subscription found; assuming standalone installation cluster_training_runtimes_test.go:245: Skipping MPI runtime 'openmpi-cuda' (covered by TestMultiNodeOpenMPITrainJob) cluster_training_runtimes_test.go:245: Skipping MPI runtime 'openmpi-cpu' (covered by TestMultiNodeOpenMPITrainJob) cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'torch-distributed-cuda130-torch211-py312' (image 'odh-th-torch-cuda-py312' already tested) cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'torch-distributed-rocm714-torch211-py312' (image 'odh-th-torch-rocm-py312' already tested) cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'torch-distributed-cpu-torch211-py312' (image 'odh-th-torch-cpu-py312' already tested) cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'training-hub' (image 'odh-th-torch-cuda-py312' already tested) cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'training-hub-cpu' (image 'odh-th-torch-cpu-py312' already tested) cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'training-hub-rocm' (image 'odh-th-torch-rocm-py312' already tested) cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'training-hub-th09-cuda130-torch211-py312' (image 'odh-th-torch-cuda-py312' already tested) cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'training-hub-th09-cpu-torch211-py312' (image 'odh-th-torch-cpu-py312' already tested) cluster_training_runtimes_test.go:253: Skipping ClusterTrainingRuntime 'training-hub-th09-rocm714-torch211-py312' (image 'odh-th-torch-rocm-py312' already tested) cluster_training_runtimes_test.go:249: Skipping speculator runtime 'vllm-extract-cuda' (covered by speculator tests) cluster_training_runtimes_test.go:249: Skipping speculator runtime 'speculator-model-opt-cuda' (covered by speculator tests) cluster_training_runtimes_test.go:290: All TrainJobs with expected ClusterTrainingRuntimes completed successfully !!! test.go:171: Retrieving Pod Container test-ns-vl8sj/test-trainjob-4m5rt-node-0-0-6lm68/node logs test.go:171: Retrieving Pod Container test-ns-d4qcn/test-trainjob-tcmjs-node-0-0-4dv56/node logs test.go:171: Retrieving Pod Container test-ns-dm4m5/test-trainjob-26qsb-node-0-0-9v7nh/node logs --- PASS: TestRunTrainJobWithDefaultClusterTrainingRuntimes (465.98s) === RUN TestJobSetWorkflow jobset_workflow_test.go:45: Created PersistentVolumeClaim test-ns-7v4rh/pvc-vs2kt successfully utils_runtimes.go:183: Using image from ClusterTrainingRuntime "torch-distributed-cpu": quay.io/opendatahub/odh-th-torch-cpu-py312:odh-pr-d8eab1e44478edf743d16ef448a1f75ca1f49c92 jobset_workflow_test.go:52: Created TrainingRuntime test-ns-7v4rh/test-trainingruntime-z7m49 jobset_workflow_test.go:55: Created TrainJob test-ns-7v4rh/test-trainjob-tgjmb jobset_workflow_test.go:61: JobSet created with 3 replicated jobs (dataset-initializer, model-initializer, node) jobset_workflow_test.go:64: Monitoring job execution ... jobset_workflow_test.go:64: dataset-initializer job is created: test-trainjob-tgjmb-dataset-initializer-0 test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestJobSetWorkflow3818631682 jobset_workflow_test.go:64: dataset-initializer job is completed: test-trainjob-tgjmb-dataset-initializer-0 jobset_workflow_test.go:64: model-initializer job is created: test-trainjob-tgjmb-model-initializer-0 jobset_workflow_test.go:64: model-initializer job is completed: test-trainjob-tgjmb-model-initializer-0 jobset_workflow_test.go:64: node job is created: test-trainjob-tgjmb-node-0 jobset_workflow_test.go:64: Sequential job execution is verified successfully: dataset-initializer → model-initializer → node jobset_workflow_test.go:69: TrainJob test-ns-7v4rh/test-trainjob-tgjmb completed test.go:171: Retrieving Pod Container test-ns-7v4rh/test-trainjob-tgjmb-dataset-initializer-0-0-56h2w/dataset-initializer logs test.go:171: Retrieving Pod Container test-ns-7v4rh/test-trainjob-tgjmb-model-initializer-0-0-xfxv6/model-initializer logs test.go:171: Retrieving Pod Container test-ns-7v4rh/test-trainjob-tgjmb-node-0-0-62sqf/node logs --- PASS: TestJobSetWorkflow (104.65s) === RUN TestFailedJobSetWorkflow test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestFailedJobSetWorkflow (0.00s) === RUN TestKubeflowSdk fashion_mnist_tests.go:56: Created User RoleBinding test-ns-v6k8g/rb-kmnws successfully fashion_mnist_tests.go:57: Created User RoleBinding test-ns-v6k8g/rb-bh6t2 successfully fashion_mnist_tests.go:57: Created ClusterRole /clusterrole-bc6vw successfully fashion_mnist_tests.go:57: Created User ClusterRoleBinding crb-9828s successfully fashion_mnist_tests.go:64: Created ConfigMap test-ns-v6k8g/config-x7jzl successfully fashion_mnist_tests.go:82: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX. fashion_mnist_tests.go:84: Created PersistentVolumeClaim test-ns-v6k8g/pvc-jm5sw successfully fashion_mnist_tests.go:93: OpenShift API URL: https://a9c6bc033741d43fbb900995d9543326-1387338057e92d6c.elb.us-east-2.amazonaws.com:6443 dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/opendatahub/odh-th-torch-cpu-py312:odh-pr-d8eab1e44478edf743d16ef448a1f75ca1f49c92 notebook.go:44: Created test Deployment test-ns-v6k8g/dw-deployment-xm5gt test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdk2834857468 test.go:171: Retrieving Pod Container test-ns-v6k8g/dw-deployment-xm5gt-5579899b56-gjn2m/notebook logs --- PASS: TestKubeflowSdk (52.98s) === RUN TestKubeflowSdkKueueIntegration kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK kueue_operator.go:227: Kueue CR 'cluster' exists kueue_operator.go:229: Waiting for Kueue CR to be ready... kueue_operator.go:236: Kueue CR is ready kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR... kueue_operator.go:243: TrainJob framework is present in Kueue CR kueue_operator.go:239: Verifying Deployment framework is present in Kueue CR... kueue_operator.go:243: Deployment framework is present in Kueue CR fashion_mnist_tests.go:148: Created Kueue-managed namespace: test-ns-7sbf4 fashion_mnist_tests.go:153: Created User RoleBinding test-ns-7sbf4/rb-mfhk5 successfully fashion_mnist_tests.go:154: Created User RoleBinding test-ns-7sbf4/rb-g7xlk successfully fashion_mnist_tests.go:154: Created ClusterRole /clusterrole-95pjk successfully fashion_mnist_tests.go:154: Created User ClusterRoleBinding crb-bdmhh successfully fashion_mnist_tests.go:157: Created Kueue ResourceFlavor rf-x6lnq successfully fashion_mnist_tests.go:184: Created Kueue ClusterQueue cq-jj7lj successfully fashion_mnist_tests.go:189: Created Kueue LocalQueue test-ns-7sbf4/lq-886bl successfully fashion_mnist_tests.go:190: Created custom LocalQueue lq-886bl for TrainJob fashion_mnist_tests.go:197: Created ConfigMap test-ns-7sbf4/config-8gjg7 successfully fashion_mnist_tests.go:215: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX. fashion_mnist_tests.go:217: Created PersistentVolumeClaim test-ns-7sbf4/pvc-82c7q successfully fashion_mnist_tests.go:226: OpenShift API URL: https://a9c6bc033741d43fbb900995d9543326-1387338057e92d6c.elb.us-east-2.amazonaws.com:6443 dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/opendatahub/odh-th-torch-cpu-py312:odh-pr-d8eab1e44478edf743d16ef448a1f75ca1f49c92 notebook.go:44: Created test Deployment test-ns-7sbf4/dw-deployment-s5xvc fashion_mnist_tests.go:271: Verifying SDK-submitted TrainJob has custom queue label: lq-886bl test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration2929072595 fashion_mnist_tests.go:280: SDK-submitted TrainJob has kueue label: kueue.x-k8s.io/queue-name=lq-886bl fashion_mnist_tests.go:283: Verifying Kueue Workloads: Deployment and TrainJob on custom queue... fashion_mnist_tests.go:297: Kueue Workload admitted successfully for SDK-submitted TrainJob test.go:171: Retrieving Pod Container test-ns-7sbf4/dw-deployment-s5xvc-6d67f9b9f6-cfktd/notebook logs --- PASS: TestKubeflowSdkKueueIntegration (51.93s) === RUN TestKubeflowSdkOpenMPICudaKueueIntegration test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestKubeflowSdkOpenMPICudaKueueIntegration (0.00s) === RUN TestSftTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestSftTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestOsftTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestOsftTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestLoraTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestLoraTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestGrpoTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestGrpoTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestSftCallbacksTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestSftCallbacksTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestLoraCallbacksTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestLoraCallbacksTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestOsftCallbacksTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestOsftCallbacksTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestOsftTrainingHubMultiNodeMultiGPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestOsftTrainingHubMultiNodeMultiGPU (0.00s) === RUN TestLoraTrainingHubMultiNodeMultiGPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestLoraTrainingHubMultiNodeMultiGPU (0.00s) === RUN TestSftTrainingHubMultiNodeMultiGPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestSftTrainingHubMultiNodeMultiGPU (0.00s) === RUN TestRhaiTrainingProgressionCPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestRhaiTrainingProgressionCPU (0.00s) === RUN TestRhaiJitCheckpointingCPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestRhaiJitCheckpointingCPU (0.00s) === RUN TestRhaiFeaturesCPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestRhaiFeaturesCPU (0.00s) === RUN TestRhaiTrainingProgressionCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiTrainingProgressionCuda (0.00s) === RUN TestRhaiJitCheckpointingCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiJitCheckpointingCuda (0.00s) === RUN TestRhaiFeaturesCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiFeaturesCuda (0.00s) === RUN TestRhaiTrainingProgressionRocm test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiTrainingProgressionRocm (0.00s) === RUN TestRhaiJitCheckpointingRocm test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiJitCheckpointingRocm (0.00s) === RUN TestRhaiFeaturesRocm test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiFeaturesRocm (0.00s) === RUN TestRhaiTrainingProgressionMultiGpuCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiTrainingProgressionMultiGpuCuda (0.00s) === RUN TestRhaiJitCheckpointingMultiGpuCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiJitCheckpointingMultiGpuCuda (0.00s) === RUN TestRhaiFeaturesMultiGpuCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiFeaturesMultiGpuCuda (0.00s) === RUN TestRhaiTrainingProgressionMultiGpuRocm test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiTrainingProgressionMultiGpuRocm (0.00s) === RUN TestRhaiJitCheckpointingMultiGpuRocm test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiJitCheckpointingMultiGpuRocm (0.00s) === RUN TestRhaiFeaturesMultiGpuRocm test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiFeaturesMultiGpuRocm (0.00s) === RUN TestTrainingFailureScenarios test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestTrainingFailureScenarios (0.00s) === RUN TestTorchrunTrainingFailure test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestTorchrunTrainingFailure (0.00s) === RUN TestRhaiS3CheckpointingCPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestRhaiS3CheckpointingCPU (0.00s) === RUN TestRhaiS3FsdpFullStateCheckpointingCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3FsdpFullStateCheckpointingCuda (0.00s) === RUN TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda (0.00s) === RUN TestRhaiS3FsdpSharedStateCheckpointingCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3FsdpSharedStateCheckpointingCuda (0.00s) === RUN TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda (0.00s) === RUN TestRhaiS3DeepspeedStage0CheckpointingCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3DeepspeedStage0CheckpointingCuda (0.00s) === RUN TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda (0.00s) === RUN TestRhaiSpeculatorPipelineSingleGpuCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiSpeculatorPipelineSingleGpuCuda (0.00s) === RUN TestRhaiSpeculatorPipelineMultiGpuCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiSpeculatorPipelineMultiGpuCuda (0.00s) === RUN TestRhaiSpeculatorOfflineSingleGpuCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiSpeculatorOfflineSingleGpuCuda (0.00s) === RUN TestRhaiSpeculatorOfflineMultiGpuCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiSpeculatorOfflineMultiGpuCuda (0.00s) === RUN TestRhaiSpeculatorOnlineSingleGpuCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiSpeculatorOnlineSingleGpuCuda (0.00s) === RUN TestRhaiSpeculatorOnlineMultiGpuCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiSpeculatorOnlineMultiGpuCuda (0.00s) === RUN TestDraFieldsOnTrainJobReachPods test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke' --- SKIP: TestDraFieldsOnTrainJobReachPods (0.00s) === RUN TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 (0.00s) === RUN TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda (0.00s) === RUN TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda (0.00s) === RUN TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda (0.00s) === RUN TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda (0.00s) === RUN TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm (0.00s) === RUN TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm (0.00s) === RUN TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm (0.00s) === RUN TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm (0.00s) === RUN TestKueueWorkloadPreemptionSuspendsTrainJob kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK kueue_operator.go:227: Kueue CR 'cluster' exists kueue_operator.go:229: Waiting for Kueue CR to be ready... kueue_operator.go:236: Kueue CR is ready kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR... kueue_operator.go:243: TrainJob framework is present in Kueue CR trainer_kueue_integration_test.go:67: Created Kueue-managed namespace: test-ns-8vdb5 trainer_kueue_integration_test.go:70: Created Kueue ResourceFlavor rf-9q6h6 successfully trainer_kueue_integration_test.go:72: Created Kueue ClusterQueue cq-rmfk4 successfully trainer_kueue_integration_test.go:96: Created Kueue LocalQueue test-ns-8vdb5/lq-6wqjg successfully trainer_kueue_integration_test.go:123: Created TrainJob: test-preemption-trainjob-ct8vf trainer_kueue_integration_test.go:133: Workload 'trainjob-test-preemption-trainjob-ct8vf-b468f' is admitted trainer_kueue_integration_test.go:137: Verifying TrainJob pods are running... test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueWorkloadPreemptionSuspendsTrainJob1472904835 trainer_kueue_integration_test.go:147: TrainJob 'test-preemption-trainjob-ct8vf' pods are running trainer_kueue_integration_test.go:150: User is preempting workload 'trainjob-test-preemption-trainjob-ct8vf-b468f' now ... trainer_kueue_integration_test.go:158: Workload 'trainjob-test-preemption-trainjob-ct8vf-b468f' is preempted trainer_kueue_integration_test.go:164: Workload 'trainjob-test-preemption-trainjob-ct8vf-b468f' is evicted trainer_kueue_integration_test.go:170: TrainJob 'test-preemption-trainjob-ct8vf' is suspended trainer_kueue_integration_test.go:172: Workload preemption successfully suspended TrainJob 'test-preemption-trainjob-ct8vf'! test.go:171: Retrieving Pod Container test-ns-8vdb5/test-preemption-trainjob-ct8vf-node-0-0-rhq6h/node logs --- PASS: TestKueueWorkloadPreemptionSuspendsTrainJob (4.49s) === RUN TestKueueWorkloadInadmissibleWithNonExistentLocalQueue test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestKueueWorkloadInadmissibleWithNonExistentLocalQueue (0.00s) === RUN TestKueueSchedulingReachesTrainJobPods kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK kueue_operator.go:227: Kueue CR 'cluster' exists kueue_operator.go:229: Waiting for Kueue CR to be ready... kueue_operator.go:236: Kueue CR is ready kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR... kueue_operator.go:243: TrainJob framework is present in Kueue CR trainer_kueue_integration_test.go:251: Created Kueue-managed namespace: test-ns-mmt5n trainer_kueue_integration_test.go:259: Created Kueue ResourceFlavor rf-q2rlq successfully trainer_kueue_integration_test.go:274: Created Kueue ClusterQueue cq-tm5jh successfully trainer_kueue_integration_test.go:298: Created Kueue LocalQueue test-ns-mmt5n/lq-f2lkt successfully trainer_kueue_integration_test.go:332: Created TrainJob: test-runtimepatches-trainjob-5rcbk trainer_kueue_integration_test.go:340: Workload admitted trainer_kueue_integration_test.go:365: JobSet "test-runtimepatches-trainjob-5rcbk" pod templates carry the ResourceFlavor's scheduling directives test.go:171: Retrieving Pod Container test-ns-mmt5n/test-runtimepatches-trainjob-5rcbk-node-0-0-xj7sx/node logs test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueSchedulingReachesTrainJobPods389156908 --- PASS: TestKueueSchedulingReachesTrainJobPods (1.37s) === RUN TestSetupUpgradeTrainJob trainer_kueue_upgrade_training_test.go:83: Skip due to issue RHOAIENG-48867 --- SKIP: TestSetupUpgradeTrainJob (0.00s) === RUN TestRunUpgradeTrainJob trainer_kueue_upgrade_training_test.go:158: Skip due to issue RHOAIENG-48867 --- SKIP: TestRunUpgradeTrainJob (0.00s) === RUN TestSetupSpecificRuntimeUpgradeTrainJob trainer_kueue_upgrade_training_test.go:205: Skip due to issue RHOAIENG-48867 --- SKIP: TestSetupSpecificRuntimeUpgradeTrainJob (0.00s) === RUN TestRunSpecificRuntimeUpgradeTrainJob trainer_kueue_upgrade_training_test.go:295: Skip due to issue RHOAIENG-48867 --- SKIP: TestRunSpecificRuntimeUpgradeTrainJob (0.00s) === RUN TestSetupCustomRuntimeUpgradeTrainJob test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade' --- SKIP: TestSetupCustomRuntimeUpgradeTrainJob (0.00s) === RUN TestRunCustomRuntimeUpgradeTrainJob test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade' --- SKIP: TestRunCustomRuntimeUpgradeTrainJob (0.00s) === RUN TestTrainerSecureServing test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestTrainerSecureServing (0.00s) === RUN TestTrainerPrometheusScrape test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestTrainerPrometheusScrape (0.00s) === RUN TestMultiNodeOpenMPICpuTrainJob trainer_mpi_test.go:51: Created ConfigMap test-ns-s5xsf/config-ggl4r successfully utils_runtimes.go:183: Using image from ClusterTrainingRuntime "openmpi-cpu": quay.io/opendatahub/odh-th-torch-cpu-py312:odh-stable test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestMultiNodeOpenMPICpuTrainJob1208262718 test.go:171: Retrieving Pod Container test-ns-s5xsf/test-mpi-trainjob-2c4nm-launcher-0-0-7w6xf/node logs test.go:171: Retrieving Pod Container test-ns-s5xsf/test-mpi-trainjob-2c4nm-node-0-0-vlngw/node logs --- PASS: TestMultiNodeOpenMPICpuTrainJob (148.51s) === RUN TestMultiNodeOpenMPITrainJob test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestMultiNodeOpenMPITrainJob (0.00s) === RUN TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics (0.00s) === RUN TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth (0.00s) === RUN TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer (0.00s) === RUN TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort (0.00s) === RUN TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort (0.00s) === RUN TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' --- SKIP: TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort (0.00s) === RUN TestOpenMPICudaTrainJobKueueIntegration test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestOpenMPICudaTrainJobKueueIntegration (0.00s) === RUN TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate (0.00s) === RUN TestSftStockTrlSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestSftStockTrlSingleNodeSingleGPU (0.00s) === RUN TestSftStockTrlSingleNodeSingleGPUWithTorchRocm test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestSftStockTrlSingleNodeSingleGPUWithTorchRocm (0.00s) === RUN TestKubeflowTrainerSmoke test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke' --- SKIP: TestKubeflowTrainerSmoke (0.00s) === RUN TestTrainerTLSProfileWatcher test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier3' --- SKIP: TestTrainerTLSProfileWatcher (0.00s) === RUN TestSetupTrainingRuntime test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade' --- SKIP: TestSetupTrainingRuntime (0.00s) === RUN TestVerifyTrainingRuntime test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade' --- SKIP: TestVerifyTrainingRuntime (0.00s) === RUN TestSetupSleepTrainJob test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade' --- SKIP: TestSetupSleepTrainJob (0.00s) === RUN TestVerifySleepTrainJob test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade' --- SKIP: TestVerifySleepTrainJob (0.00s) PASS TearDown: Skipping Kueue teardown because no DataScienceCluster is installed ok github.com/opendatahub-io/distributed-workloads/tests/trainer 829.996s === Skipped === SKIP: tests/trainer TestDefaultClusterTrainingRuntimes (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke' === SKIP: tests/trainer TestOpenMPICudaClusterTrainingRuntime (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke' === SKIP: tests/trainer TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke' === SKIP: tests/trainer TestFailedJobSetWorkflow (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestKubeflowSdkOpenMPICudaKueueIntegration (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestSftTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestOsftTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestLoraTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestGrpoTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestSftCallbacksTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestLoraCallbacksTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestOsftCallbacksTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestOsftTrainingHubMultiNodeMultiGPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestLoraTrainingHubMultiNodeMultiGPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestSftTrainingHubMultiNodeMultiGPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiTrainingProgressionCPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestRhaiJitCheckpointingCPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestRhaiFeaturesCPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestRhaiTrainingProgressionCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiJitCheckpointingCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiFeaturesCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiTrainingProgressionRocm (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestRhaiJitCheckpointingRocm (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestRhaiFeaturesRocm (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestRhaiTrainingProgressionMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiJitCheckpointingMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiFeaturesMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiTrainingProgressionMultiGpuRocm (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestRhaiJitCheckpointingMultiGpuRocm (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestRhaiFeaturesMultiGpuRocm (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestTrainingFailureScenarios (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestTorchrunTrainingFailure (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiS3CheckpointingCPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestRhaiS3FsdpFullStateCheckpointingCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiS3FsdpSharedStateCheckpointingCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiS3DeepspeedStage0CheckpointingCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiSpeculatorPipelineSingleGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiSpeculatorPipelineMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiSpeculatorOfflineSingleGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiSpeculatorOfflineMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiSpeculatorOnlineSingleGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiSpeculatorOnlineMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestDraFieldsOnTrainJobReachPods (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke' === SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestKueueWorkloadInadmissibleWithNonExistentLocalQueue (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestSetupUpgradeTrainJob (0.00s) trainer_kueue_upgrade_training_test.go:83: Skip due to issue RHOAIENG-48867 === SKIP: tests/trainer TestRunUpgradeTrainJob (0.00s) trainer_kueue_upgrade_training_test.go:158: Skip due to issue RHOAIENG-48867 === SKIP: tests/trainer TestSetupSpecificRuntimeUpgradeTrainJob (0.00s) trainer_kueue_upgrade_training_test.go:205: Skip due to issue RHOAIENG-48867 === SKIP: tests/trainer TestRunSpecificRuntimeUpgradeTrainJob (0.00s) trainer_kueue_upgrade_training_test.go:295: Skip due to issue RHOAIENG-48867 === SKIP: tests/trainer TestSetupCustomRuntimeUpgradeTrainJob (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade' === SKIP: tests/trainer TestRunCustomRuntimeUpgradeTrainJob (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade' === SKIP: tests/trainer TestTrainerSecureServing (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestTrainerPrometheusScrape (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestMultiNodeOpenMPITrainJob (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier2' === SKIP: tests/trainer TestOpenMPICudaTrainJobKueueIntegration (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestSftStockTrlSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestSftStockTrlSingleNodeSingleGPUWithTorchRocm (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestKubeflowTrainerSmoke (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Smoke' === SKIP: tests/trainer TestTrainerTLSProfileWatcher (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Tier3' === SKIP: tests/trainer TestSetupTrainingRuntime (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade' === SKIP: tests/trainer TestVerifyTrainingRuntime (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade' === SKIP: tests/trainer TestSetupSleepTrainJob (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Pre-Upgrade' === SKIP: tests/trainer TestVerifySleepTrainJob (0.00s) test_tag.go:37: Test tier 'Tier1' doesn't match expected tier 'Post-Upgrade' DONE 88 tests, 81 skipped in 829.996s