Initial Kueue managementState: === RUN TestDefaultClusterTrainingRuntimes test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Smoke' --- SKIP: TestDefaultClusterTrainingRuntimes (0.00s) === RUN TestOpenMPICudaClusterTrainingRuntime test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Smoke' --- SKIP: TestOpenMPICudaClusterTrainingRuntime (0.00s) === RUN TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Smoke' --- SKIP: TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes (0.00s) === RUN TestRunTrainJobWithDefaultClusterTrainingRuntimes test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' --- SKIP: TestRunTrainJobWithDefaultClusterTrainingRuntimes (0.00s) === RUN TestJobSetWorkflow test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' --- SKIP: TestJobSetWorkflow (0.00s) === RUN TestFailedJobSetWorkflow jobset_workflow_test.go:80: Created PersistentVolumeClaim test-ns-99z99/pvc-wsfxp successfully utils_runtimes.go:183: Using image from ClusterTrainingRuntime "torch-distributed-cpu": quay.io/opendatahub/odh-th-torch-cpu-py312:odh-pr-aadbb1f7aa4c719b22de3a1fb7c866c7a536ec59 jobset_workflow_test.go:87: Created TrainingRuntime test-ns-99z99/test-trainingruntime-4jhk5 jobset_workflow_test.go:90: Created TrainJob test-ns-99z99/test-trainjob-fail-2nt7v test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestFailedJobSetWorkflow3993347292 jobset_workflow_test.go:99: JobSet failed as expected jobset_workflow_test.go:104: TrainJob failed as expected test.go:171: Retrieving Pod Container test-ns-99z99/test-trainjob-fail-2nt7v-dataset-initializer-0-0-9z68d/dataset-initializer logs --- PASS: TestFailedJobSetWorkflow (11.56s) === RUN TestKubeflowSdk test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' --- SKIP: TestKubeflowSdk (0.00s) === RUN TestKubeflowSdkKueueIntegration test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' --- SKIP: TestKubeflowSdkKueueIntegration (0.00s) === RUN TestKubeflowSdkOpenMPICudaKueueIntegration test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestKubeflowSdkOpenMPICudaKueueIntegration (0.00s) === RUN TestSftTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestSftTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestOsftTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestOsftTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestLoraTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestLoraTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestGrpoTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestGrpoTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestSftCallbacksTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestSftCallbacksTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestLoraCallbacksTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestLoraCallbacksTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestOsftCallbacksTrainingHubSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestOsftCallbacksTrainingHubSingleNodeSingleGPU (0.00s) === RUN TestOsftTrainingHubMultiNodeMultiGPU test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestOsftTrainingHubMultiNodeMultiGPU (0.00s) === RUN TestLoraTrainingHubMultiNodeMultiGPU test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestLoraTrainingHubMultiNodeMultiGPU (0.00s) === RUN TestSftTrainingHubMultiNodeMultiGPU test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestSftTrainingHubMultiNodeMultiGPU (0.00s) === RUN TestRhaiTrainingProgressionCPU rhai_features_tests.go:258: Created User RoleBinding test-ns-42bwn/rb-8dbgb successfully rhai_features_tests.go:259: Created User RoleBinding test-ns-42bwn/rb-vxq9g successfully rhai_features_tests.go:259: Created ClusterRole /clusterrole-lq7qc successfully rhai_features_tests.go:259: Created User ClusterRoleBinding crb-65th7 successfully rhai_features_tests.go:273: Created ConfigMap test-ns-42bwn/config-rspqp successfully rhai_features_tests.go:276: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX. rhai_features_tests.go:278: Created PersistentVolumeClaim test-ns-42bwn/pvc-5v4gd successfully rhai_features_tests.go:319: Using internal endpoint for in-cluster pods: http://s4.s4.svc.cluster.local:7480 (external: https://s4-api-s4.apps.90c857067d838b886141.prod.konflux-ocp-ci.dev) rhai_features_tests.go:364: HuggingFace mode: S3 not configured for models/datasets, will download from HF Hub rhai_features_tests.go:402: Using Red Hat PyPI index for cpu (kubeflow not on public PyPI) rhai_features_tests.go:416: OpenShift API URL: https://a03be5fe4cfdd424ebe4e86a59a0b82a-5f3e8419f7ca2d38.elb.us-east-2.amazonaws.com:6443 rhai_features_tests.go:444: Feature config: ProgressionTracking=true, JitCheckpoint=false, Accelerator=cpu, NumNodes=1, NumGpusPerNode=1, Notebook=rhai_features.ipynb dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/opendatahub/odh-th-torch-cpu-py312:odh-pr-aadbb1f7aa4c719b22de3a1fb7c866c7a536ec59 notebook.go:44: Created test Deployment test-ns-42bwn/dw-deployment-65bxj test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestRhaiTrainingProgressionCPU2808146728 rhai_features_tests.go:487: TrainJob created: c2f980578811 rhai_features_tests.go:491: Verifying TrainJob annotations match expected config... rhai_features_tests.go:491: progression-tracking: expected=true, actual=true rhai_features_tests.go:491: TrainJob annotations match expected config rhai_features_tests.go:501: TrainJob c2f980578811 completed successfully rhai_features_tests.go:510: Verifying progression tracking features... rhai_features_tests.go:513: Found 1 training pod(s) for TrainJob c2f980578811 rhai_features_tests.go:513: Pod c2f980578811-node-0-0-9ltgx termination message: {"progressPercentage": 100, "estimatedRemainingSeconds": 0, "currentStep": 100, "totalSteps": 100, "currentEpoch": 5.0, "totalEpochs": 5, "trainMetrics": {"loss": 2.0777618408203127, "grad_norm": 7.419865131378174, "learning_rate": 5.000000000000001e-07, "throughput_samples_sec": 0.834, "train_loss": 2.4961353015899657}, "evalMetrics": {"eval_loss": 2.5037848949432373, "eval_runtime": 5.4916, "eval_samples_per_second": 3.642, "eval_steps_per_second": 0.546}} rhai_features_tests.go:513: trainMetrics: map[grad_norm:7.419865131378174 learning_rate:5.000000000000001e-07 loss:2.0777618408203127 throughput_samples_sec:0.834 train_loss:2.4961353015899657] rhai_features_tests.go:513: evalMetrics: map[eval_loss:2.5037848949432373 eval_runtime:5.4916 eval_samples_per_second:3.642 eval_steps_per_second:0.546] rhai_features_tests.go:513: Termination message verified for pod c2f980578811-node-0-0-9ltgx rhai_features_tests.go:518: progression-tracking annotation is 'true' rhai_features_tests.go:522: metrics-port annotation is '28080' rhai_features_tests.go:526: metrics-poll-interval annotation is '5' rhai_features_tests.go:531: Verifying training completion via pod termination message (authoritative source)... rhai_features_tests.go:549: Found 100% progress in termination message for pod c2f980578811-node-0-0-9ltgx rhai_features_tests.go:562: Waiting for trainerStatus annotation to reach 100% progress... rhai_features_tests.go:581: trainerStatus: {"progressPercentage":100,"estimatedRemainingSeconds":0,"currentStep":100,"totalSteps":100,"currentEpoch":5,"totalEpochs":5,"trainMetrics":{"grad_norm":7.419865131378174,"learning_rate":5.000000000000001e-7,"loss":2.0777618408203127},"evalMetrics":{"eval_loss":2.509791135787964,"eval_runtime":5.5173,"eval_samples_per_second":3.625,"eval_steps_per_second":0.544},"lastUpdatedTime":"2026-10-30T11:21:48Z"} rhai_features_tests.go:586: progressPercentage: 100% rhai_features_tests.go:590: currentStep: 100/100 rhai_features_tests.go:594: currentEpoch: 5/5 rhai_features_tests.go:599: estimatedRemainingSeconds: 0 rhai_features_tests.go:601: Progression tracking verification passed! rhai_features_tests.go:613: All RHAI features tests passed! test.go:171: Retrieving Pod Container test-ns-42bwn/c2f980578811-node-0-0-9ltgx/node logs test.go:171: Retrieving Pod Container test-ns-42bwn/dw-deployment-65bxj-5b666d8b55-qd2h9/notebook logs --- PASS: TestRhaiTrainingProgressionCPU (514.31s) === RUN TestRhaiJitCheckpointingCPU rhai_features_tests.go:258: Created User RoleBinding test-ns-zqs2h/rb-c9sr4 successfully rhai_features_tests.go:259: Created User RoleBinding test-ns-zqs2h/rb-mt6c5 successfully rhai_features_tests.go:259: Created ClusterRole /clusterrole-xnk77 successfully rhai_features_tests.go:259: Created User ClusterRoleBinding crb-kx6bt successfully rhai_features_tests.go:273: Created ConfigMap test-ns-zqs2h/config-2b6m6 successfully rhai_features_tests.go:276: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX. rhai_features_tests.go:278: Created PersistentVolumeClaim test-ns-zqs2h/pvc-5nnlc successfully rhai_features_tests.go:319: Using internal endpoint for in-cluster pods: http://s4.s4.svc.cluster.local:7480 (external: https://s4-api-s4.apps.90c857067d838b886141.prod.konflux-ocp-ci.dev) rhai_features_tests.go:364: HuggingFace mode: S3 not configured for models/datasets, will download from HF Hub rhai_features_tests.go:402: Using Red Hat PyPI index for cpu (kubeflow not on public PyPI) rhai_features_tests.go:416: OpenShift API URL: https://a03be5fe4cfdd424ebe4e86a59a0b82a-5f3e8419f7ca2d38.elb.us-east-2.amazonaws.com:6443 rhai_features_tests.go:444: Feature config: ProgressionTracking=false, JitCheckpoint=true, Accelerator=cpu, NumNodes=1, NumGpusPerNode=1, Notebook=rhai_features.ipynb dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/opendatahub/odh-th-torch-cpu-py312:odh-pr-aadbb1f7aa4c719b22de3a1fb7c866c7a536ec59 notebook.go:44: Created test Deployment test-ns-zqs2h/dw-deployment-sffsq test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestRhaiJitCheckpointingCPU2401623353 rhai_features_tests.go:487: TrainJob created: x835b0f7f181 rhai_features_tests.go:491: Verifying TrainJob annotations match expected config... rhai_features_tests.go:491: progression-tracking: expected=empty/false, actual= rhai_features_tests.go:491: TrainJob annotations match expected config rhai_features_tests.go:495: Running JIT checkpoint suspend/resume test... rhai_features_tests.go:496: Starting JIT checkpoint verification for TrainJob x835b0f7f181 (checkpoint_dir=/workspace/checkpoints) rhai_features_tests.go:496: Step 1: Waiting for training pods to start... rhai_features_tests.go:496: Training pods are running rhai_features_tests.go:496: Waiting for training to complete at least 2 epochs (checking logs)... rhai_features_tests.go:1057: Epoch 2 (>= 2) detected in pod x835b0f7f181-node-0-0-crtcd logs rhai_features_tests.go:496: At least 2 epochs completed - ready to suspend rhai_features_tests.go:496: Step 2: Suspending TrainJob to trigger checkpoint save... rhai_features_tests.go:496: TrainJob x835b0f7f181 suspended rhai_features_tests.go:496: Step 3: Waiting for training pods to fully terminate... rhai_features_tests.go:496: All training pods terminated - checkpoint should be saved rhai_features_tests.go:496: Step 4: Verifying job is suspended (not completed or failed)... rhai_features_tests.go:496: TrainJob is suspended rhai_features_tests.go:496: Step 6: Resuming TrainJob to verify checkpoint restore... rhai_features_tests.go:496: TrainJob x835b0f7f181 resumed rhai_features_tests.go:496: Waiting for new training pods to start... rhai_features_tests.go:496: New training pods started rhai_features_tests.go:496: Step 8: Waiting for training to complete after resume... rhai_features_tests.go:496: TrainJob completed successfully after checkpoint restore rhai_features_tests.go:496: Step 9: Verifying checkpoint was loaded from logs... rhai_features_tests.go:496: Checkpoint resume verified in pod x835b0f7f181-node-0-0-5txq8: Found latest checkpoint: rhai_features_tests.go:496: JIT checkpoint verification completed successfully! rhai_features_tests.go:604: Verifying progression tracking is disabled... rhai_features_tests.go:610: trainerStatus annotation is empty as expected rhai_features_tests.go:613: All RHAI features tests passed! test.go:171: Retrieving Pod Container test-ns-zqs2h/dw-deployment-sffsq-86b764b69f-j5vfh/notebook logs test.go:171: Retrieving Pod Container test-ns-zqs2h/x835b0f7f181-node-0-0-5txq8/node logs --- PASS: TestRhaiJitCheckpointingCPU (587.90s) === RUN TestRhaiFeaturesCPU rhai_features_tests.go:258: Created User RoleBinding test-ns-vbf7g/rb-xg8wt successfully rhai_features_tests.go:259: Created User RoleBinding test-ns-vbf7g/rb-vx44h successfully rhai_features_tests.go:259: Created ClusterRole /clusterrole-7d2kg successfully rhai_features_tests.go:259: Created User ClusterRoleBinding crb-czflk successfully rhai_features_tests.go:273: Created ConfigMap test-ns-vbf7g/config-jrm2s successfully rhai_features_tests.go:276: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX. rhai_features_tests.go:278: Created PersistentVolumeClaim test-ns-vbf7g/pvc-m995c successfully rhai_features_tests.go:319: Using internal endpoint for in-cluster pods: http://s4.s4.svc.cluster.local:7480 (external: https://s4-api-s4.apps.90c857067d838b886141.prod.konflux-ocp-ci.dev) rhai_features_tests.go:364: HuggingFace mode: S3 not configured for models/datasets, will download from HF Hub rhai_features_tests.go:402: Using Red Hat PyPI index for cpu (kubeflow not on public PyPI) rhai_features_tests.go:416: OpenShift API URL: https://a03be5fe4cfdd424ebe4e86a59a0b82a-5f3e8419f7ca2d38.elb.us-east-2.amazonaws.com:6443 rhai_features_tests.go:444: Feature config: ProgressionTracking=true, JitCheckpoint=true, Accelerator=cpu, NumNodes=1, NumGpusPerNode=1, Notebook=rhai_features.ipynb dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/opendatahub/odh-th-torch-cpu-py312:odh-pr-aadbb1f7aa4c719b22de3a1fb7c866c7a536ec59 notebook.go:44: Created test Deployment test-ns-vbf7g/dw-deployment-d6lrp test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestRhaiFeaturesCPU1604892096 rhai_features_tests.go:487: TrainJob created: j27692c258fd rhai_features_tests.go:491: Verifying TrainJob annotations match expected config... rhai_features_tests.go:491: progression-tracking: expected=true, actual=true rhai_features_tests.go:491: TrainJob annotations match expected config rhai_features_tests.go:495: Running JIT checkpoint suspend/resume test... rhai_features_tests.go:496: Starting JIT checkpoint verification for TrainJob j27692c258fd (checkpoint_dir=/workspace/checkpoints) rhai_features_tests.go:496: Step 1: Waiting for training pods to start... rhai_features_tests.go:496: Training pods are running rhai_features_tests.go:496: Waiting for training to complete at least 2 epochs (checking logs)... rhai_features_tests.go:1057: Epoch 2 (>= 2) detected in pod j27692c258fd-node-0-0-979rz logs rhai_features_tests.go:496: At least 2 epochs completed - ready to suspend rhai_features_tests.go:496: Waiting for progression tracking to be captured by operator... rhai_features_tests.go:496: Progression captured by operator rhai_features_tests.go:496: Step 2: Suspending TrainJob to trigger checkpoint save... rhai_features_tests.go:496: TrainJob j27692c258fd suspended rhai_features_tests.go:496: Step 3: Waiting for training pods to fully terminate... rhai_features_tests.go:496: All training pods terminated - checkpoint should be saved rhai_features_tests.go:496: Step 4: Verifying job is suspended (not completed or failed)... rhai_features_tests.go:496: TrainJob is suspended rhai_features_tests.go:496: Step 5: Reading progression state after suspend... rhai_features_tests.go:496: Pre-resume state: epoch=2.00, progress=40% rhai_features_tests.go:496: Step 6: Resuming TrainJob to verify checkpoint restore... rhai_features_tests.go:496: TrainJob j27692c258fd resumed rhai_features_tests.go:496: Waiting for new training pods to start... rhai_features_tests.go:496: New training pods started rhai_features_tests.go:496: Step 7: Waiting for progress update after resume... rhai_features_tests.go:496: Post-resume state: epoch=2.00, progress=40% rhai_features_tests.go:496: Checkpoint verification: Progress preserved after resume rhai_features_tests.go:496: Step 8: Waiting for training to complete after resume... rhai_features_tests.go:496: TrainJob completed successfully after checkpoint restore rhai_features_tests.go:496: Step 9: Verifying checkpoint was loaded from logs... rhai_features_tests.go:496: Checkpoint resume verified in pod j27692c258fd-node-0-0-jlzcx: Found latest checkpoint: rhai_features_tests.go:496: JIT checkpoint verification completed successfully! rhai_features_tests.go:510: Verifying progression tracking features... rhai_features_tests.go:513: Found 1 training pod(s) for TrainJob j27692c258fd rhai_features_tests.go:513: Pod j27692c258fd-node-0-0-jlzcx termination message: {"progressPercentage": 100, "estimatedRemainingSeconds": 0, "currentStep": 100, "totalSteps": 100, "currentEpoch": 5.0, "totalEpochs": 5, "trainMetrics": {"loss": 2.064496612548828, "grad_norm": 7.317352294921875, "learning_rate": 5.000000000000001e-07, "throughput_samples_sec": 1.286, "train_loss": 1.344347324371338}, "evalMetrics": {"eval_loss": 2.4996681213378906, "eval_runtime": 5.5141, "eval_samples_per_second": 3.627, "eval_steps_per_second": 0.544}} rhai_features_tests.go:513: trainMetrics: map[grad_norm:7.317352294921875 learning_rate:5.000000000000001e-07 loss:2.064496612548828 throughput_samples_sec:1.286 train_loss:1.344347324371338] rhai_features_tests.go:513: evalMetrics: map[eval_loss:2.4996681213378906 eval_runtime:5.5141 eval_samples_per_second:3.627 eval_steps_per_second:0.544] rhai_features_tests.go:513: Termination message verified for pod j27692c258fd-node-0-0-jlzcx rhai_features_tests.go:518: progression-tracking annotation is 'true' rhai_features_tests.go:522: metrics-port annotation is '28080' rhai_features_tests.go:526: metrics-poll-interval annotation is '5' rhai_features_tests.go:531: Verifying training completion via pod termination message (authoritative source)... rhai_features_tests.go:549: Found 100% progress in termination message for pod j27692c258fd-node-0-0-jlzcx rhai_features_tests.go:562: Waiting for trainerStatus annotation to reach 100% progress... rhai_features_tests.go:581: trainerStatus: {"progressPercentage":100,"estimatedRemainingSeconds":0,"currentStep":100,"totalSteps":100,"currentEpoch":5,"totalEpochs":5,"trainMetrics":{"grad_norm":7.317352294921875,"learning_rate":5.000000000000001e-7,"loss":2.064496612548828},"evalMetrics":{"eval_loss":2.5052921772003174,"eval_runtime":5.4459,"eval_samples_per_second":3.672,"eval_steps_per_second":0.551},"lastUpdatedTime":"2026-10-30T11:41:21Z"} rhai_features_tests.go:586: progressPercentage: 100% rhai_features_tests.go:590: currentStep: 100/100 rhai_features_tests.go:594: currentEpoch: 5/5 rhai_features_tests.go:599: estimatedRemainingSeconds: 0 rhai_features_tests.go:601: Progression tracking verification passed! rhai_features_tests.go:613: All RHAI features tests passed! test.go:171: Retrieving Pod Container test-ns-vbf7g/dw-deployment-d6lrp-6d5b8db8f5-fp2sq/notebook logs test.go:171: Retrieving Pod Container test-ns-vbf7g/j27692c258fd-node-0-0-jlzcx/node logs --- PASS: TestRhaiFeaturesCPU (599.17s) === RUN TestRhaiTrainingProgressionCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiTrainingProgressionCuda (0.00s) === RUN TestRhaiJitCheckpointingCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiJitCheckpointingCuda (0.00s) === RUN TestRhaiFeaturesCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiFeaturesCuda (0.00s) === RUN TestRhaiTrainingProgressionRocm test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiTrainingProgressionRocm (0.00s) === RUN TestRhaiJitCheckpointingRocm test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiJitCheckpointingRocm (0.00s) === RUN TestRhaiFeaturesRocm test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiFeaturesRocm (0.00s) === RUN TestRhaiTrainingProgressionMultiGpuCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiTrainingProgressionMultiGpuCuda (0.00s) === RUN TestRhaiJitCheckpointingMultiGpuCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiJitCheckpointingMultiGpuCuda (0.00s) === RUN TestRhaiFeaturesMultiGpuCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiFeaturesMultiGpuCuda (0.00s) === RUN TestRhaiTrainingProgressionMultiGpuRocm test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiTrainingProgressionMultiGpuRocm (0.00s) === RUN TestRhaiJitCheckpointingMultiGpuRocm test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiJitCheckpointingMultiGpuRocm (0.00s) === RUN TestRhaiFeaturesMultiGpuRocm test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestRhaiFeaturesMultiGpuRocm (0.00s) === RUN TestTrainingFailureScenarios test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestTrainingFailureScenarios (0.00s) === RUN TestTorchrunTrainingFailure test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestTorchrunTrainingFailure (0.00s) === RUN TestRhaiS3CheckpointingCPU rhai_features_tests.go:189: Using folder s3://training-kubeflow-e2e/checkpoints-TestRhaiS3CheckpointingCPU-1793360505243158983 for checkpoints rhai_features_tests.go:258: Created User RoleBinding test-ns-cgqx9/rb-gclmb successfully rhai_features_tests.go:259: Created User RoleBinding test-ns-cgqx9/rb-cxbm4 successfully rhai_features_tests.go:259: Created ClusterRole /clusterrole-v7hgn successfully rhai_features_tests.go:259: Created User ClusterRoleBinding crb-tjkjt successfully rhai_features_tests.go:273: Created ConfigMap test-ns-cgqx9/config-g46mx successfully rhai_features_tests.go:276: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX. rhai_features_tests.go:278: Created PersistentVolumeClaim test-ns-cgqx9/pvc-wnktw successfully rhai_features_tests.go:319: Using internal endpoint for in-cluster pods: http://s4.s4.svc.cluster.local:7480 (external: https://s4-api-s4.apps.90c857067d838b886141.prod.konflux-ocp-ci.dev) rhai_features_tests.go:364: HuggingFace mode: S3 not configured for models/datasets, will download from HF Hub rhai_features_tests.go:382: Created Secret test-ns-cgqx9/secret-v7zhf successfully rhai_features_tests.go:383: Created Data Connection secret: secret-v7zhf for cloud checkpoint storage rhai_features_tests.go:389: Data Connection configured for cloud checkpointing: s3://training-kubeflow-e2e/checkpoints-TestRhaiS3CheckpointingCPU-1793360505243158983/checkpoints rhai_features_tests.go:402: Using Red Hat PyPI index for cpu (kubeflow not on public PyPI) rhai_features_tests.go:416: OpenShift API URL: https://a03be5fe4cfdd424ebe4e86a59a0b82a-5f3e8419f7ca2d38.elb.us-east-2.amazonaws.com:6443 rhai_features_tests.go:444: Feature config: ProgressionTracking=false, JitCheckpoint=true, Accelerator=cpu, NumNodes=1, NumGpusPerNode=1, Notebook=rhai_features.ipynb dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/opendatahub/odh-th-torch-cpu-py312:odh-pr-aadbb1f7aa4c719b22de3a1fb7c866c7a536ec59 notebook.go:44: Created test Deployment test-ns-cgqx9/dw-deployment-pnr9c test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestRhaiS3CheckpointingCPU2913839822 rhai_features_tests.go:487: TrainJob created: ca87ddc200f0 rhai_features_tests.go:491: Verifying TrainJob annotations match expected config... rhai_features_tests.go:491: progression-tracking: expected=empty/false, actual= rhai_features_tests.go:491: TrainJob annotations match expected config rhai_features_tests.go:495: Running JIT checkpoint suspend/resume test... rhai_features_tests.go:496: Starting JIT checkpoint verification for TrainJob ca87ddc200f0 (checkpoint_dir=s3://training-kubeflow-e2e/checkpoints-TestRhaiS3CheckpointingCPU-1793360505243158983/checkpoints) rhai_features_tests.go:496: Step 1: Waiting for training pods to start... rhai_features_tests.go:496: Training pods are running rhai_features_tests.go:496: Waiting for training to complete at least 2 epochs (checking logs)... rhai_features_tests.go:1057: Epoch 2 (>= 2) detected in pod ca87ddc200f0-node-0-0-4hwrj logs rhai_features_tests.go:496: At least 2 epochs completed - ready to suspend rhai_features_tests.go:496: Step 1b: Verifying cloud checkpoint upload is working... rhai_features_tests.go:780: SDK save_strategy override confirmed in pod logs rhai_features_tests.go:798: Cloud checkpoint upload confirmed in pod logs rhai_features_tests.go:496: Cloud checkpoint upload verified in logs - verifying checkpoints exist in S3... rhai_features_tests.go:814: Checkpoints verified in S3: s3://training-kubeflow-e2e/checkpoints-TestRhaiS3CheckpointingCPU-1793360505243158983/checkpoints rhai_features_tests.go:496: Cloud checkpoint upload verified - checkpoints confirmed in S3 storage rhai_features_tests.go:496: Step 2: Suspending TrainJob to trigger checkpoint save... rhai_features_tests.go:496: TrainJob ca87ddc200f0 suspended rhai_features_tests.go:496: Step 3: Waiting for training pods to fully terminate... rhai_features_tests.go:496: All training pods terminated - checkpoint should be saved rhai_features_tests.go:496: Step 4: Verifying job is suspended (not completed or failed)... rhai_features_tests.go:496: TrainJob is suspended rhai_features_tests.go:496: Step 6: Resuming TrainJob to verify checkpoint restore... rhai_features_tests.go:496: TrainJob ca87ddc200f0 resumed rhai_features_tests.go:496: Waiting for new training pods to start... rhai_features_tests.go:496: New training pods started rhai_features_tests.go:496: Step 8: Waiting for training to complete after resume... rhai_features_tests.go:496: TrainJob completed successfully after checkpoint restore rhai_features_tests.go:496: Step 9: Verifying checkpoint was loaded from logs... rhai_features_tests.go:496: Checkpoint resume verified in pod ca87ddc200f0-node-0-0-vwktw: Download complete rhai_features_tests.go:496: JIT checkpoint verification completed successfully! rhai_features_tests.go:604: Verifying progression tracking is disabled... rhai_features_tests.go:610: trainerStatus annotation is empty as expected rhai_features_tests.go:613: All RHAI features tests passed! rhai_features_tests.go:185: Test folder deleted: s3://training-kubeflow-e2e/checkpoints-TestRhaiS3CheckpointingCPU-1793360505243158983 test.go:171: Retrieving Pod Container test-ns-cgqx9/ca87ddc200f0-node-0-0-vwktw/node logs test.go:171: Retrieving Pod Container test-ns-cgqx9/dw-deployment-pnr9c-859c49ff56-z89gs/notebook logs --- PASS: TestRhaiS3CheckpointingCPU (593.39s) === RUN TestRhaiS3FsdpFullStateCheckpointingCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3FsdpFullStateCheckpointingCuda (0.00s) === RUN TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda (0.00s) === RUN TestRhaiS3FsdpSharedStateCheckpointingCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3FsdpSharedStateCheckpointingCuda (0.00s) === RUN TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda (0.00s) === RUN TestRhaiS3DeepspeedStage0CheckpointingCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3DeepspeedStage0CheckpointingCuda (0.00s) === RUN TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda (0.00s) === RUN TestRhaiSpeculatorPipelineSingleGpuCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiSpeculatorPipelineSingleGpuCuda (0.00s) === RUN TestRhaiSpeculatorPipelineMultiGpuCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiSpeculatorPipelineMultiGpuCuda (0.00s) === RUN TestRhaiSpeculatorOfflineSingleGpuCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiSpeculatorOfflineSingleGpuCuda (0.00s) === RUN TestRhaiSpeculatorOfflineMultiGpuCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiSpeculatorOfflineMultiGpuCuda (0.00s) === RUN TestRhaiSpeculatorOnlineSingleGpuCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiSpeculatorOnlineSingleGpuCuda (0.00s) === RUN TestRhaiSpeculatorOnlineMultiGpuCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestRhaiSpeculatorOnlineMultiGpuCuda (0.00s) === RUN TestDraFieldsOnTrainJobReachPods test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Smoke' --- SKIP: TestDraFieldsOnTrainJobReachPods (0.00s) === RUN TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 utils_runtimes.go:183: Using image from ClusterTrainingRuntime "torch-distributed-cpu": quay.io/opendatahub/odh-th-torch-cpu-py312:odh-pr-aadbb1f7aa4c719b22de3a1fb7c866c7a536ec59 kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK kueue_operator.go:227: Kueue CR 'cluster' exists kueue_operator.go:229: Waiting for Kueue CR to be ready... kueue_operator.go:236: Kueue CR is ready kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR... kueue_operator.go:243: TrainJob framework is present in Kueue CR trainer_fashion_mnist_training_test.go:91: Created Kueue-managed namespace: test-ns-s4smk trainer_fashion_mnist_training_test.go:94: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX. trainer_fashion_mnist_training_test.go:98: Created PersistentVolumeClaim test-ns-s4smk/pvc-z4mxt successfully trainer_fashion_mnist_training_test.go:106: Created ConfigMap test-ns-s4smk/config-2t86w successfully trainer_fashion_mnist_training_test.go:109: Created Kueue ResourceFlavor rf-ggnqg successfully trainer_fashion_mnist_training_test.go:151: Created Kueue ClusterQueue cq-6sp2p successfully trainer_fashion_mnist_training_test.go:153: Created Kueue LocalQueue test-ns-s4smk/lq-bpl5g successfully trainer_fashion_mnist_training_test.go:156: Created TrainingRuntime test-ns-s4smk/test-fashion-mnist-runtime-qnkwx with dataset-initializer trainer_fashion_mnist_training_test.go:159: S3/Minio configuration incomplete or not provided, dataset-initializer will use public mirrors trainer_fashion_mnist_training_test.go:159: Created TrainJob test-ns-s4smk/test-fashion-mnist-trainjob-mfhd5 trainer_fashion_mnist_training_test.go:169: Kueue Workload admitted trainer_fashion_mnist_training_test.go:172: Verifying JobSet creation with replicated jobs... trainer_fashion_mnist_training_test.go:176: JobSet created with 2 replicated jobs (dataset-initializer, node) trainer_fashion_mnist_training_test.go:178: Checking JobSet status... test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda284235444420 trainer_fashion_mnist_training_test.go:182: JobSet marked as completed trainer_fashion_mnist_training_test.go:187: TrainJob test-ns-s4smk/test-fashion-mnist-trainjob-mfhd5 completed test.go:171: Retrieving Pod Container test-ns-s4smk/test-fashion-mnist-trainjob-mfhd5-dataset-initializer-0-0-7z6lh/dataset-initializer logs test.go:171: Retrieving Pod Container test-ns-s4smk/test-fashion-mnist-trainjob-mfhd5-node-0-0-6hpz4/node logs test.go:171: Retrieving Pod Container test-ns-s4smk/test-fashion-mnist-trainjob-mfhd5-node-0-1-bg8d2/node logs --- PASS: TestPyTorchDDPMultiNodeMultiCPUWithTorchCuda28 (119.02s) === RUN TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda (0.00s) === RUN TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda (0.00s) === RUN TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda (0.00s) === RUN TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda (0.00s) === RUN TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm (0.00s) === RUN TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm (0.00s) === RUN TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm (0.00s) === RUN TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm (0.00s) === RUN TestKueueWorkloadPreemptionSuspendsTrainJob test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' --- SKIP: TestKueueWorkloadPreemptionSuspendsTrainJob (0.00s) === RUN TestKueueWorkloadInadmissibleWithNonExistentLocalQueue kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK kueue_operator.go:227: Kueue CR 'cluster' exists kueue_operator.go:229: Waiting for Kueue CR to be ready... kueue_operator.go:236: Kueue CR is ready kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR... kueue_operator.go:243: TrainJob framework is present in Kueue CR trainer_kueue_integration_test.go:182: Created Kueue-managed namespace: test-ns-p8mpn trainer_kueue_integration_test.go:187: Creating TrainJob with non-existent LocalQueue... trainer_kueue_integration_test.go:213: TrainJob created: test-non-existent-localqueue-trainjob-49nsb trainer_kueue_integration_test.go:216: Verifying Workload is Inadmissible due to non-existent LocalQueue... trainer_kueue_integration_test.go:230: Workload 'trainjob-test-non-existent-localqueue-trainjob-49nsb-deb4e' is Inadmissible as LocalQueue non-existent-queue doesn't exist trainer_kueue_integration_test.go:236: TrainJob 'test-non-existent-localqueue-trainjob-49nsb' is suspended as expected trainer_kueue_integration_test.go:238: Non-existent LocalQueue causes Workload Inadmissible - verified successfully! test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKueueWorkloadInadmissibleWithNonExistentLocalQueue4203520458 --- PASS: TestKueueWorkloadInadmissibleWithNonExistentLocalQueue (1.32s) === RUN TestKueueSchedulingReachesTrainJobPods test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' --- SKIP: TestKueueSchedulingReachesTrainJobPods (0.00s) === RUN TestSetupUpgradeTrainJob trainer_kueue_upgrade_training_test.go:83: Skip due to issue RHOAIENG-48867 --- SKIP: TestSetupUpgradeTrainJob (0.00s) === RUN TestRunUpgradeTrainJob trainer_kueue_upgrade_training_test.go:158: Skip due to issue RHOAIENG-48867 --- SKIP: TestRunUpgradeTrainJob (0.00s) === RUN TestSetupSpecificRuntimeUpgradeTrainJob trainer_kueue_upgrade_training_test.go:205: Skip due to issue RHOAIENG-48867 --- SKIP: TestSetupSpecificRuntimeUpgradeTrainJob (0.00s) === RUN TestRunSpecificRuntimeUpgradeTrainJob trainer_kueue_upgrade_training_test.go:295: Skip due to issue RHOAIENG-48867 --- SKIP: TestRunSpecificRuntimeUpgradeTrainJob (0.00s) === RUN TestSetupCustomRuntimeUpgradeTrainJob test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Pre-Upgrade' --- SKIP: TestSetupCustomRuntimeUpgradeTrainJob (0.00s) === RUN TestRunCustomRuntimeUpgradeTrainJob trainer_kueue_upgrade_training_test.go:458: Skip due to issue RHOAIENG-87834 --- SKIP: TestRunCustomRuntimeUpgradeTrainJob (0.00s) === RUN TestTrainerSecureServing dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub trainer_metrics_test.go:38: Trainer controller pod kubeflow-trainer-controller-manager-84b74c44f8-fgw9c is Ready trainer_metrics_test.go:39: Checking Trainer metrics service opendatahub/kubeflow-trainer-controller-manager trainer_metrics_test.go:63: Created TokenRequest opendatahub/default successfully trainer_metrics_test.go:73: Created TokenRequest openshift-monitoring/prometheus-k8s successfully trainer_metrics_test.go:77: Validated Trainer HTTPS metrics authentication for service opendatahub/kubeflow-trainer-controller-manager via pod kubeflow-trainer-controller-manager-84b74c44f8-fgw9c E1030 11:53:39.655187 19218 portforward.go:525] "Error copying from local connection to remote stream" err="writeto tcp4 127.0.0.1:39105->127.0.0.1:34110: read tcp4 127.0.0.1:39105->127.0.0.1:34110: read: connection reset by peer" logger="UnhandledError" localPort=39105 remotePort=8443 --- PASS: TestTrainerSecureServing (0.68s) === RUN TestTrainerPrometheusScrape dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub prometheus_api_client.go:48: Created TokenRequest openshift-monitoring/prometheus-k8s successfully trainer_metrics_test.go:99: Waiting for Prometheus to discover Trainer ServiceMonitor target opendatahub/kubeflow-trainer-controller-manager trainer_metrics_test.go:119: Prometheus target is healthy: scrapeURL=https://10.134.0.16:8443/metrics trainer_metrics_test.go:143: Prometheus returned up=1 and certwatcher metrics for Trainer target --- PASS: TestTrainerPrometheusScrape (0.33s) === RUN TestMultiNodeOpenMPICpuTrainJob test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' --- SKIP: TestMultiNodeOpenMPICpuTrainJob (0.00s) === RUN TestMultiNodeOpenMPITrainJob test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestMultiNodeOpenMPITrainJob (0.00s) === RUN TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub trainer_network_policy_test.go:48: Trainer controller pod kubeflow-trainer-controller-manager-84b74c44f8-fgw9c is Ready dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub trainer_network_policy_test.go:49: Trainer controller pod kubeflow-trainer-controller-manager-84b74c44f8-fgw9c is Ready trainer_network_policy_test.go:49: Created Pod test-ns-tndss/trainer-netpol-curl-dcn8r successfully test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics803245190 trainer_network_policy_test.go:49: curl probe pod test-ns-tndss/trainer-netpol-curl-dcn8r target=https://10.134.0.16:8443 exit=28; logs: * Trying 10.134.0.16:8443... * Connection timed out after 5001 milliseconds * Closing connection 0 curl: (28) Connection timed out after 5001 milliseconds CURL_TIMING time_connect=0.000000 time_total=5.001462 test.go:171: Retrieving Pod Container test-ns-tndss/trainer-netpol-curl-dcn8r/curl logs --- PASS: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToMetrics (8.37s) === RUN TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub trainer_network_policy_test.go:54: Trainer controller pod kubeflow-trainer-controller-manager-84b74c44f8-fgw9c is Ready dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub trainer_network_policy_test.go:55: Trainer controller pod kubeflow-trainer-controller-manager-84b74c44f8-fgw9c is Ready trainer_network_policy_test.go:55: Created Pod test-ns-l2qns/trainer-netpol-curl-dqhbj successfully test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth2533166972 trainer_network_policy_test.go:55: curl probe pod test-ns-l2qns/trainer-netpol-curl-dqhbj target=http://10.134.0.16:8081 exit=28; logs: * Trying 10.134.0.16:8081... * Connection timed out after 5000 milliseconds * Closing connection 0 curl: (28) Connection timed out after 5000 milliseconds CURL_TIMING time_connect=0.000000 time_total=5.000926 test.go:171: Retrieving Pod Container test-ns-l2qns/trainer-netpol-curl-dqhbj/curl logs --- PASS: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToHealth (8.36s) === RUN TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub trainer_network_policy_test.go:60: Trainer controller pod kubeflow-trainer-controller-manager-84b74c44f8-fgw9c is Ready dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub trainer_network_policy_test.go:61: Trainer controller pod kubeflow-trainer-controller-manager-84b74c44f8-fgw9c is Ready trainer_network_policy_test.go:61: Created Pod test-ns-tqqc6/trainer-netpol-curl-bqpbc successfully test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer3921306230 trainer_network_policy_test.go:61: curl probe pod test-ns-tqqc6/trainer-netpol-curl-bqpbc target=https://10.134.0.16:10443 exit=28; logs: * Trying 10.134.0.16:10443... * Connection timed out after 5001 milliseconds * Closing connection 0 curl: (28) Connection timed out after 5001 milliseconds CURL_TIMING time_connect=0.000000 time_total=5.001396 test.go:171: Retrieving Pod Container test-ns-tqqc6/trainer-netpol-curl-bqpbc/curl logs --- PASS: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToStatusServer (6.32s) === RUN TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: opendatahub trainer_network_policy_test.go:68: Trainer controller pod kubeflow-trainer-controller-manager-84b74c44f8-fgw9c is Ready trainer_network_policy_test.go:68: Created Pod test-ns-fljc2/trainer-netpol-curl-bjrmj successfully test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort4186352790 trainer_network_policy_test.go:68: curl probe pod test-ns-fljc2/trainer-netpol-curl-bjrmj target=https://10.134.0.16:31415 exit=28; logs: * Trying 10.134.0.16:31415... * Connection timed out after 5001 milliseconds * Closing connection 0 curl: (28) Connection timed out after 5001 milliseconds CURL_TIMING time_connect=0.000000 time_total=5.001490 test.go:171: Retrieving Pod Container test-ns-fljc2/trainer-netpol-curl-bjrmj/curl logs --- PASS: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToUnpublishedPort (6.25s) === RUN TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort utils_runtimes.go:183: Using image from ClusterTrainingRuntime "torch-distributed-cpu": quay.io/opendatahub/odh-th-torch-cpu-py312:odh-pr-aadbb1f7aa4c719b22de3a1fb7c866c7a536ec59 trainer_network_policy_test.go:131: Created CPU TrainJob test-ns-64dlw/trainer-netpol-workload-lxrkh using runtime "torch-distributed-cpu" test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort2374894048 trainer_network_policy_test.go:151: TrainJob pod test-ns-64dlw/trainer-netpol-workload-lxrkh-node-0-0-98dtl completed its local HTTP readiness request on port 18080 trainer_network_policy_test.go:153: Created Pod test-ns-j4khd/trainer-netpol-curl-7m8hn successfully trainer_network_policy_test.go:153: curl probe pod test-ns-j4khd/trainer-netpol-curl-7m8hn target=http://10.132.0.69:18080 exit=28; logs: * Trying 10.132.0.69:18080... * Connection timed out after 5000 milliseconds * Closing connection 0 curl: (28) Connection timed out after 5000 milliseconds CURL_TIMING time_connect=0.000000 time_total=4.999848 test.go:171: Retrieving Pod Container test-ns-j4khd/trainer-netpol-curl-7m8hn/curl logs test.go:171: Retrieving Pod Container test-ns-64dlw/trainer-netpol-workload-lxrkh-node-0-0-98dtl/node logs --- PASS: TestTrainerNetworkPolicyBlocksIngressFromOtherNamespacesToWorkloadPort (11.51s) === RUN TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort utils_runtimes.go:183: Using image from ClusterTrainingRuntime "torch-distributed-cpu": quay.io/opendatahub/odh-th-torch-cpu-py312:odh-pr-aadbb1f7aa4c719b22de3a1fb7c866c7a536ec59 trainer_network_policy_test.go:131: Created CPU TrainJob test-ns-g4dw9/trainer-netpol-workload-h27rg using runtime "torch-distributed-cpu" test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort2712989371 trainer_network_policy_test.go:151: TrainJob pod test-ns-g4dw9/trainer-netpol-workload-h27rg-node-0-0-sjjgf completed its local HTTP readiness request on port 18080 trainer_network_policy_test.go:153: Created Pod test-ns-g4dw9/trainer-netpol-curl-l6flp successfully trainer_network_policy_test.go:153: curl probe pod test-ns-g4dw9/trainer-netpol-curl-l6flp target=http://10.132.0.71:18080 exit=28; logs: * Trying 10.132.0.71:18080... * Connection timed out after 5001 milliseconds * Closing connection 0 curl: (28) Connection timed out after 5001 milliseconds CURL_TIMING time_connect=0.000000 time_total=5.001503 test.go:171: Retrieving Pod Container test-ns-g4dw9/trainer-netpol-curl-l6flp/curl logs test.go:171: Retrieving Pod Container test-ns-g4dw9/trainer-netpol-workload-h27rg-node-0-0-sjjgf/node logs --- PASS: TestTrainerNetworkPolicyBlocksIngressFromOtherWorkloadsToWorkloadPort (9.43s) === RUN TestOpenMPICudaTrainJobKueueIntegration test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestOpenMPICudaTrainJobKueueIntegration (0.00s) === RUN TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate (0.00s) === RUN TestSftStockTrlSingleNodeSingleGPU test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' --- SKIP: TestSftStockTrlSingleNodeSingleGPU (0.00s) === RUN TestSftStockTrlSingleNodeSingleGPUWithTorchRocm test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' --- SKIP: TestSftStockTrlSingleNodeSingleGPUWithTorchRocm (0.00s) === RUN TestKubeflowTrainerSmoke test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Smoke' --- SKIP: TestKubeflowTrainerSmoke (0.00s) === RUN TestTrainerTLSProfileWatcher test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier3' --- SKIP: TestTrainerTLSProfileWatcher (0.00s) === RUN TestSetupTrainingRuntime test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Pre-Upgrade' --- SKIP: TestSetupTrainingRuntime (0.00s) === RUN TestVerifyTrainingRuntime test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Post-Upgrade' --- SKIP: TestVerifyTrainingRuntime (0.00s) === RUN TestSetupSleepTrainJob test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Pre-Upgrade' --- SKIP: TestSetupSleepTrainJob (0.00s) === RUN TestVerifySleepTrainJob test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Post-Upgrade' --- SKIP: TestVerifySleepTrainJob (0.00s) PASS TearDown: Skipping Kueue teardown because no DataScienceCluster is installed ok github.com/opendatahub-io/distributed-workloads/tests/trainer 2478.001s === Skipped === SKIP: tests/trainer TestDefaultClusterTrainingRuntimes (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Smoke' === SKIP: tests/trainer TestOpenMPICudaClusterTrainingRuntime (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Smoke' === SKIP: tests/trainer TestDefaultTrainingHubRuntimesMatchDefaultClusterRuntimes (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Smoke' === SKIP: tests/trainer TestRunTrainJobWithDefaultClusterTrainingRuntimes (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' === SKIP: tests/trainer TestJobSetWorkflow (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' === SKIP: tests/trainer TestKubeflowSdk (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' === SKIP: tests/trainer TestKubeflowSdkKueueIntegration (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' === SKIP: tests/trainer TestKubeflowSdkOpenMPICudaKueueIntegration (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestSftTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestOsftTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestLoraTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestGrpoTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestSftCallbacksTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestLoraCallbacksTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestOsftCallbacksTrainingHubSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestOsftTrainingHubMultiNodeMultiGPU (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestLoraTrainingHubMultiNodeMultiGPU (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestSftTrainingHubMultiNodeMultiGPU (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiTrainingProgressionCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiJitCheckpointingCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiFeaturesCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiTrainingProgressionRocm (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestRhaiJitCheckpointingRocm (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestRhaiFeaturesRocm (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestRhaiTrainingProgressionMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiJitCheckpointingMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiFeaturesMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiTrainingProgressionMultiGpuRocm (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestRhaiJitCheckpointingMultiGpuRocm (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestRhaiFeaturesMultiGpuRocm (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestTrainingFailureScenarios (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestTorchrunTrainingFailure (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiS3FsdpFullStateCheckpointingCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiS3FsdpFullStateCheckpointingMultiProcessCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiS3FsdpSharedStateCheckpointingCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiS3FsdpSharedStateCheckpointingMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiS3DeepspeedStage0CheckpointingCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiS3DeepspeedStage0CheckpointingMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiSpeculatorPipelineSingleGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiSpeculatorPipelineMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiSpeculatorOfflineSingleGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiSpeculatorOfflineMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiSpeculatorOnlineSingleGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestRhaiSpeculatorOnlineMultiGpuCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestDraFieldsOnTrainJobReachPods (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Smoke' === SKIP: tests/trainer TestPyTorchDDPSingleNodeSingleGPUWithTorchCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestPyTorchDDPSingleNodeMultiGPUWithTorchCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestPyTorchDDPMultiNodeSingleGPUWithTorchCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiGPUWithTorchCuda (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestPyTorchDDPSingleNodeSingleGPUWithTorchRocm (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestPyTorchDDPSingleNodeMultiGPUWithTorchRocm (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestPyTorchDDPMultiNodeSingleGPUWithTorchRocm (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestPyTorchDDPMultiNodeMultiGPUWithTorchRocm (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestKueueWorkloadPreemptionSuspendsTrainJob (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' === SKIP: tests/trainer TestKueueSchedulingReachesTrainJobPods (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' === SKIP: tests/trainer TestSetupUpgradeTrainJob (0.00s) trainer_kueue_upgrade_training_test.go:83: Skip due to issue RHOAIENG-48867 === SKIP: tests/trainer TestRunUpgradeTrainJob (0.00s) trainer_kueue_upgrade_training_test.go:158: Skip due to issue RHOAIENG-48867 === SKIP: tests/trainer TestSetupSpecificRuntimeUpgradeTrainJob (0.00s) trainer_kueue_upgrade_training_test.go:205: Skip due to issue RHOAIENG-48867 === SKIP: tests/trainer TestRunSpecificRuntimeUpgradeTrainJob (0.00s) trainer_kueue_upgrade_training_test.go:295: Skip due to issue RHOAIENG-48867 === SKIP: tests/trainer TestSetupCustomRuntimeUpgradeTrainJob (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Pre-Upgrade' === SKIP: tests/trainer TestRunCustomRuntimeUpgradeTrainJob (0.00s) trainer_kueue_upgrade_training_test.go:458: Skip due to issue RHOAIENG-87834 === SKIP: tests/trainer TestMultiNodeOpenMPICpuTrainJob (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier1' === SKIP: tests/trainer TestMultiNodeOpenMPITrainJob (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestOpenMPICudaTrainJobKueueIntegration (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestOpenMPICudaTrainJobKueueWorkloadDeactivateReactivate (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestSftStockTrlSingleNodeSingleGPU (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-CUDA' === SKIP: tests/trainer TestSftStockTrlSingleNodeSingleGPUWithTorchRocm (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'KFTO-ROCm' === SKIP: tests/trainer TestKubeflowTrainerSmoke (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Smoke' === SKIP: tests/trainer TestTrainerTLSProfileWatcher (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Tier3' === SKIP: tests/trainer TestSetupTrainingRuntime (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Pre-Upgrade' === SKIP: tests/trainer TestVerifyTrainingRuntime (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Post-Upgrade' === SKIP: tests/trainer TestSetupSleepTrainJob (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Pre-Upgrade' === SKIP: tests/trainer TestVerifySleepTrainJob (0.00s) test_tag.go:37: Test tier 'Tier2' doesn't match expected tier 'Post-Upgrade' DONE 88 tests, 73 skipped in 2478.001s