=== RUN TestSetupPytorchjob kueue_operator.go:109: SetupKueue: Setting kueue to Unmanaged managementState in DataScienceCluster... kueue_operator.go:122: SetupKueue: Kueue is set to Unmanaged managementState successfully kueue_operator.go:216: Kueue CR 'cluster' exists kueue_operator.go:218: Waiting for Kueue CR to be ready... kueue_operator.go:225: Kueue CR is ready kueue_operator.go:228: Verifying PyTorchJob framework is present in Kueue CR... kueue_operator.go:232: PyTorchJob framework is present in Kueue CR kfto_kueue_mnist_upgrade_training_test.go:52: test-kfto-upgrade namespace doesn't exist. Creating ... kfto_kueue_mnist_upgrade_training_test.go:53: Created Kueue-managed namespace: test-kfto-upgrade kfto_kueue_mnist_upgrade_training_test.go:66: Created ConfigMap test-kfto-upgrade/config-kvfbv successfully kfto_kueue_mnist_upgrade_training_test.go:72: Applied Kueue ResourceFlavor rf-upgrade successfully kfto_kueue_mnist_upgrade_training_test.go:93: Applied Kueue ClusterQueue cq-upgrade successfully kfto_kueue_mnist_upgrade_training_test.go:102: Applied Kueue LocalQueue test-kfto-upgrade/lq-upgrade successfully environment.go:117: Operator RELATED_IMAGE RELATED_IMAGE_ODH_TRAINING_CUDA124_TORCH25_PY311_IMAGE not found, using default: quay.io/rhoai/odh-training-cuda124-torch25-py311-rhel9:rhoai-3.4 kfto_kueue_mnist_upgrade_training_test.go:297: Skipped usage of S3 storage bucket, because required environment variables aren't provided! Required environment variables : AWS_DEFAULT_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_STORAGE_BUCKET, AWS_STORAGE_BUCKET_MNIST_DIR kfto_kueue_mnist_upgrade_training_test.go:301: Unexpected error: <*errors.StatusError | 0x21d649ea4000>: the server could not find the requested resource (post pytorchjobs.kubeflow.org) { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "the server could not find the requested resource (post pytorchjobs.kubeflow.org)", Reason: "NotFound", Details: { Name: "", Group: "kubeflow.org", Kind: "pytorchjobs", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred kueue_operator.go:99: Collecting Kueue diagnostics... test.go:152: Creating ephemeral output directory as TEST_OUTPUT_DIR env variable is unset test.go:160: Output directory has been created at: /tmp/TestSetupPytorchjob4276343874 kueue_operator.go:99: Stored Kueue CR state kueue_operator.go:99: Stored DSC state kueue_operator.go:99: Stored Kueue operator logs for kueue-controller-manager-594877d585-87ggz/manager kueue_operator.go:99: Stored Kueue operator logs for kueue-controller-manager-594877d585-rhszx/manager --- FAIL: TestSetupPytorchjob (5.88s) === RUN TestRunPytorchjob kfto_kueue_mnist_upgrade_training_test.go:129: Timed out after 300.001s. The function passed to Eventually failed at /distributed-workloads/tests/common/support/pytorchjob.go:30 with: Unexpected error: <*errors.StatusError | 0x21d649b2d9a0>: the server could not find the requested resource (get pytorchjobs.kubeflow.org pytorch-upgrade) { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "the server could not find the requested resource (get pytorchjobs.kubeflow.org pytorch-upgrade)", Reason: "NotFound", Details: { Name: "pytorch-upgrade", Group: "kubeflow.org", Kind: "pytorchjobs", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred test.go:152: Creating ephemeral output directory as TEST_OUTPUT_DIR env variable is unset test.go:160: Output directory has been created at: /tmp/TestRunPytorchjob723105115 --- FAIL: TestRunPytorchjob (300.13s) === RUN TestSetupSleepPytorchjob kfto_upgrade_sleep_test.go:43: test-kfto-upgrade-sleep namespace doesn't exist. Creating ... kfto_upgrade_sleep_test.go:44: Created namespace: test-kfto-upgrade-sleep kfto_upgrade_sleep_test.go:115: Unexpected error: <*errors.StatusError | 0x21d6499e19a0>: the server could not find the requested resource (post pytorchjobs.kubeflow.org) { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "the server could not find the requested resource (post pytorchjobs.kubeflow.org)", Reason: "NotFound", Details: { Name: "", Group: "kubeflow.org", Kind: "pytorchjobs", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred --- FAIL: TestSetupSleepPytorchjob (0.06s) === RUN TestVerifySleepPytorchjob pytorchjob.go:30: Unexpected error: <*errors.StatusError | 0x21d649e99220>: the server could not find the requested resource (get pytorchjobs.kubeflow.org pytorch-upgrade-sleep) { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "the server could not find the requested resource (get pytorchjobs.kubeflow.org pytorch-upgrade-sleep)", Reason: "NotFound", Details: { Name: "pytorch-upgrade-sleep", Group: "kubeflow.org", Kind: "pytorchjobs", UID: "", Causes: [ { Type: "UnexpectedServerResponse", Message: "404 page not found", Field: "", }, ], RetryAfterSeconds: 0, }, Code: 404, }, } occurred test.go:152: Creating ephemeral output directory as TEST_OUTPUT_DIR env variable is unset test.go:160: Output directory has been created at: /tmp/TestVerifySleepPytorchjob3107621860 --- FAIL: TestVerifySleepPytorchjob (0.08s)