=== RUN TestKubeflowSdk fashion_mnist_tests.go:56: Created User RoleBinding test-ns-tdnz6/rb-xnlvv successfully fashion_mnist_tests.go:57: Created User RoleBinding test-ns-tdnz6/rb-7rdkg successfully fashion_mnist_tests.go:57: Created ClusterRole /clusterrole-chpc2 successfully fashion_mnist_tests.go:57: Created User ClusterRoleBinding crb-b5trf successfully fashion_mnist_tests.go:64: Created ConfigMap test-ns-tdnz6/config-8ngq6 successfully fashion_mnist_tests.go:82: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX. fashion_mnist_tests.go:84: Created PersistentVolumeClaim test-ns-tdnz6/pvc-n7pzh successfully fashion_mnist_tests.go:93: OpenShift API URL: https://a97f8398d4dc944878cfd92eb7b75d4e-7a4ecc4a4c388e7e.elb.us-east-2.amazonaws.com:6443 dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1140 notebook.go:44: Created test Deployment test-ns-tdnz6/dw-deployment-f8sq7 test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdk1481802450 fashion_mnist_tests.go:138: Deployment runner execution reported FAILURE Unexpected error: <*errors.errorString | 0x2fdce6f4f3d0>: pod execution failed { s: "pod execution failed", } occurred test.go:171: Retrieving Pod Container test-ns-tdnz6/dw-deployment-f8sq7-797844fc4b-7mt2w/notebook logs --- FAIL: TestKubeflowSdk (13.70s) === RUN TestKubeflowSdkKueueIntegration kueue_operator.go:105: SetupKueue: no DataScienceCluster found; using standalone RHBoK kueue_operator.go:227: Kueue CR 'cluster' exists kueue_operator.go:229: Waiting for Kueue CR to be ready... kueue_operator.go:236: Kueue CR is ready kueue_operator.go:239: Verifying TrainJob framework is present in Kueue CR... kueue_operator.go:243: TrainJob framework is present in Kueue CR kueue_operator.go:239: Verifying Deployment framework is present in Kueue CR... kueue_operator.go:243: Deployment framework is present in Kueue CR fashion_mnist_tests.go:148: Created Kueue-managed namespace: test-ns-jxhrf fashion_mnist_tests.go:153: Created User RoleBinding test-ns-jxhrf/rb-7vvlx successfully fashion_mnist_tests.go:154: Created User RoleBinding test-ns-jxhrf/rb-2vjbp successfully fashion_mnist_tests.go:154: Created ClusterRole /clusterrole-h4lq6 successfully fashion_mnist_tests.go:154: Created User ClusterRoleBinding crb-jw2pt successfully fashion_mnist_tests.go:157: Created Kueue ResourceFlavor rf-5kg2d successfully fashion_mnist_tests.go:184: Created Kueue ClusterQueue cq-ssz5w successfully fashion_mnist_tests.go:189: Created Kueue LocalQueue test-ns-jxhrf/lq-zkkb9 successfully fashion_mnist_tests.go:190: Created custom LocalQueue lq-zkkb9 for TrainJob fashion_mnist_tests.go:197: Created ConfigMap test-ns-jxhrf/config-8mpd4 successfully fashion_mnist_tests.go:215: Found NFS StorageClass 'nfs' with provisioner 'example.com/nfs' which supports RWX. fashion_mnist_tests.go:217: Created PersistentVolumeClaim test-ns-jxhrf/pvc-xgpvx successfully fashion_mnist_tests.go:226: OpenShift API URL: https://a97f8398d4dc944878cfd92eb7b75d4e-7a4ecc4a4c388e7e.elb.us-east-2.amazonaws.com:6443 dscInitialization.go:44: Using applications namespace from env var APPLICATIONS_NAMESPACE: redhat-ods-applications environment.go:170: Using notebook image from ImageStream training-hub-universal-cpu:3.6 source: quay.io/rhoai/odh-th-torch-cpu-py312-rhel9:on-pr-1140 notebook.go:44: Created test Deployment test-ns-jxhrf/dw-deployment-hcnpm fashion_mnist_tests.go:271: Verifying SDK-submitted TrainJob has custom queue label: lq-zkkb9 test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration3181399757 fashion_mnist_tests.go:272: Timed out after 1200.001s. Expected <[]v1alpha1.TrainJob | len:0, cap:0>: [] to have length 1 test.go:171: Retrieving Pod Container test-ns-jxhrf/dw-deployment-hcnpm-58d569f9dd-r225h/notebook logs kueue_operator.go:100: Collecting Kueue diagnostics... test.go:145: Creating output directory in parent directory: /workspace/artifacts-dir test.go:160: Output directory has been created at: /workspace/artifacts-dir/TestKubeflowSdkKueueIntegration1541199642 kueue_operator.go:100: Stored Kueue CR state kueue_operator.go:100: DataScienceCluster is not installed; skipping DSC diagnostics kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-6f9bf978cc-l6rsc/manager kueue_operator.go:100: Stored Kueue operator logs for kueue-controller-manager-6f9bf978cc-z8b62/manager --- FAIL: TestKubeflowSdkKueueIntegration (1200.97s)