=== RUN TestDefaultClusterTrainingRuntimes cluster_training_runtimes_test.go:39: Detected operator image prefix: quay.io/rhoai cluster_training_runtimes_test.go:68: ClusterTrainingRuntime 'torch-distributed' is present cluster_training_runtimes_test.go:85: Image referred in ClusterTrainingRuntime is registry.redhat.io/rhoai/odh-th-torch-cuda-py312-rhel9@sha256:6add65e2cc0bb72417fa3723d4fbf0ef286752913a8be831516c2571f0adffb5 cluster_training_runtimes_test.go:115: ClusterTrainingRuntime 'torch-distributed' uses expected image: registry.redhat.io/rhoai/odh-th cluster_training_runtimes_test.go:68: ClusterTrainingRuntime 'torch-distributed-cpu' is present cluster_training_runtimes_test.go:85: Image referred in ClusterTrainingRuntime is registry.redhat.io/rhoai/odh-th-torch-cpu-py312-rhel9@sha256:5d5ecdc6f94e9377f80c165d54944ea7bb062821ec78e6a3503768a6c3e04283 cluster_training_runtimes_test.go:115: ClusterTrainingRuntime 'torch-distributed-cpu' uses expected image: registry.redhat.io/rhoai/odh-th-torch-cpu-py312 cluster_training_runtimes_test.go:68: ClusterTrainingRuntime 'torch-distributed-cpu-torch211-py312' is present cluster_training_runtimes_test.go:85: Image referred in ClusterTrainingRuntime is registry.redhat.io/rhoai/odh-th-torch-cpu-py312-rhel9@sha256:5d5ecdc6f94e9377f80c165d54944ea7bb062821ec78e6a3503768a6c3e04283 cluster_training_runtimes_test.go:115: ClusterTrainingRuntime 'torch-distributed-cpu-torch211-py312' uses expected image: registry.redhat.io/rhoai/odh-th-torch-cpu-py312 cluster_training_runtimes_test.go:68: ClusterTrainingRuntime 'torch-distributed-cuda130-torch211-py312' is present cluster_training_runtimes_test.go:85: Image referred in ClusterTrainingRuntime is registry.redhat.io/rhoai/odh-th-torch-cuda-py312-rhel9@sha256:6add65e2cc0bb72417fa3723d4fbf0ef286752913a8be831516c2571f0adffb5 cluster_training_runtimes_test.go:115: ClusterTrainingRuntime 'torch-distributed-cuda130-torch211-py312' uses expected image: registry.redhat.io/rhoai/odh-th cluster_training_runtimes_test.go:68: ClusterTrainingRuntime 'torch-distributed-rocm' is present cluster_training_runtimes_test.go:85: Image referred in ClusterTrainingRuntime is registry.redhat.io/rhoai/odh-th-torch-rocm-py312-rhel9@sha256:58f0a5560cdff04b9440d608849010264f46491afc9a8fb475d0aa10b6bec810 cluster_training_runtimes_test.go:115: ClusterTrainingRuntime 'torch-distributed-rocm' uses expected image: registry.redhat.io/rhoai/odh-th-torch-rocm-py312 cluster_training_runtimes_test.go:68: ClusterTrainingRuntime 'torch-distributed-rocm714-torch211-py312' is present cluster_training_runtimes_test.go:85: Image referred in ClusterTrainingRuntime is registry.redhat.io/rhoai/odh-th-torch-rocm-py312-rhel9@sha256:58f0a5560cdff04b9440d608849010264f46491afc9a8fb475d0aa10b6bec810 cluster_training_runtimes_test.go:115: ClusterTrainingRuntime 'torch-distributed-rocm714-torch211-py312' uses expected image: registry.redhat.io/rhoai/odh-th-torch-rocm-py312 cluster_training_runtimes_test.go:68: ClusterTrainingRuntime 'training-hub' is present cluster_training_runtimes_test.go:85: Image referred in ClusterTrainingRuntime is registry.redhat.io/rhoai/odh-th-torch-cuda-py312-rhel9@sha256:6add65e2cc0bb72417fa3723d4fbf0ef286752913a8be831516c2571f0adffb5 cluster_training_runtimes_test.go:115: ClusterTrainingRuntime 'training-hub' uses expected image: registry.redhat.io/rhoai/odh-th cluster_training_runtimes_test.go:68: ClusterTrainingRuntime 'training-hub-cpu' is present cluster_training_runtimes_test.go:85: Image referred in ClusterTrainingRuntime is registry.redhat.io/rhoai/odh-th-torch-cpu-py312-rhel9@sha256:5d5ecdc6f94e9377f80c165d54944ea7bb062821ec78e6a3503768a6c3e04283 cluster_training_runtimes_test.go:115: ClusterTrainingRuntime 'training-hub-cpu' uses expected image: registry.redhat.io/rhoai/odh-th-torch-cpu-py312 cluster_training_runtimes_test.go:68: ClusterTrainingRuntime 'training-hub-rocm' is present cluster_training_runtimes_test.go:85: Image referred in ClusterTrainingRuntime is registry.redhat.io/rhoai/odh-th-torch-rocm-py312-rhel9@sha256:58f0a5560cdff04b9440d608849010264f46491afc9a8fb475d0aa10b6bec810 cluster_training_runtimes_test.go:115: ClusterTrainingRuntime 'training-hub-rocm' uses expected image: registry.redhat.io/rhoai/odh-th-torch-rocm-py312 cluster_training_runtimes_test.go:68: ClusterTrainingRuntime 'training-hub-th09-cpu-torch211-py312' is present cluster_training_runtimes_test.go:85: Image referred in ClusterTrainingRuntime is registry.redhat.io/rhoai/odh-th-torch-cpu-py312-rhel9@sha256:5d5ecdc6f94e9377f80c165d54944ea7bb062821ec78e6a3503768a6c3e04283 cluster_training_runtimes_test.go:115: ClusterTrainingRuntime 'training-hub-th09-cpu-torch211-py312' uses expected image: registry.redhat.io/rhoai/odh-th-torch-cpu-py312 cluster_training_runtimes_test.go:68: ClusterTrainingRuntime 'training-hub-th09-cuda130-torch211-py312' is present cluster_training_runtimes_test.go:85: Image referred in ClusterTrainingRuntime is registry.redhat.io/rhoai/odh-th-torch-cuda-py312-rhel9@sha256:6add65e2cc0bb72417fa3723d4fbf0ef286752913a8be831516c2571f0adffb5 cluster_training_runtimes_test.go:115: ClusterTrainingRuntime 'training-hub-th09-cuda130-torch211-py312' uses expected image: registry.redhat.io/rhoai/odh-th cluster_training_runtimes_test.go:68: ClusterTrainingRuntime 'training-hub-th09-rocm714-torch211-py312' is present cluster_training_runtimes_test.go:85: Image referred in ClusterTrainingRuntime is registry.redhat.io/rhoai/odh-th-torch-rocm-py312-rhel9@sha256:58f0a5560cdff04b9440d608849010264f46491afc9a8fb475d0aa10b6bec810 cluster_training_runtimes_test.go:115: ClusterTrainingRuntime 'training-hub-th09-rocm714-torch211-py312' uses expected image: registry.redhat.io/rhoai/odh-th-torch-rocm-py312 cluster_training_runtimes_test.go:146: Missing expected ClusterTrainingRuntimes: [vllm-extract-cuda speculator-model-opt-cuda]. These runtimes should be present on the cluster. Expected <[]string | len:2, cap:2>: [ "vllm-extract-cuda", "speculator-model-opt-cuda", ] to be empty --- FAIL: TestDefaultClusterTrainingRuntimes (1.55s)