Adding Tier1 kuberay e2e tests (including autoscaler, packages: ./test/e2erayjob ./test/e2eautoscaler) Adding Smoke kuberay e2e tests (packages: ./test/e2e) Selected tiers: Tier1 Smoke Test packages: ./test/e2erayjob ./test/e2eautoscaler ./test/e2e Running e2e tests matching: ^(TestRayJobWithClusterSelector|TestRayJob|TestRayJobSuspend|TestRayJobLightWeightMode|TestRayClusterAutoscaler|TestRayClusterAuthenticationRhoai|TestRayClusterAuthOptions|TestRayClusterRHOAIImages)$ Test timeout: 60m go: downloading github.com/onsi/gomega v1.38.3 go: downloading github.com/stretchr/testify v1.11.1 go: downloading k8s.io/api v0.35.0 go: downloading k8s.io/apimachinery v0.35.0 go: downloading k8s.io/client-go v0.35.0 go: downloading k8s.io/utils v0.0.0-20251002143259-bc988d571ff4 go: downloading sigs.k8s.io/controller-runtime v0.22.4 go: downloading github.com/openshift/api v0.0.0-20251214014457-bfa868a22401 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading sigs.k8s.io/gateway-api v1.4.1 go: downloading github.com/onsi/ginkgo/v2 v2.27.3 go: downloading github.com/prometheus/client_golang v1.23.2 go: downloading github.com/Masterminds/semver/v3 v3.4.0 go: downloading sigs.k8s.io/yaml v1.6.0 go: downloading github.com/go-logr/logr v1.4.3 go: downloading k8s.io/apiserver v0.34.3 go: downloading k8s.io/component-base v0.34.3 go: downloading github.com/google/go-cmp v0.7.0 go: downloading k8s.io/klog/v2 v2.130.1 go: downloading sigs.k8s.io/randfill v1.0.0 go: downloading github.com/cert-manager/cert-manager v1.18.2 go: downloading github.com/orcaman/concurrent-map/v2 v2.0.1 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/kube-openapi v0.0.0-20251125145642-4e65d59e963e go: downloading sigs.k8s.io/structured-merge-diff/v6 v6.3.1 go: downloading github.com/hashicorp/golang-lru/v2 v2.0.7 go: downloading github.com/smallnest/chanx v1.2.0 go: downloading sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 go: downloading github.com/google/gnostic-models v0.7.1 go: downloading google.golang.org/protobuf v1.36.11 go: downloading github.com/evanphx/json-patch/v5 v5.9.11 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/client_model v0.6.2 go: downloading github.com/prometheus/common v0.67.4 go: downloading github.com/prometheus/procfs v0.19.2 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading golang.org/x/net v0.48.0 go: downloading github.com/spf13/pflag v1.0.10 go: downloading golang.org/x/term v0.38.0 go: downloading github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading go.yaml.in/yaml/v2 v2.4.3 go: downloading go.yaml.in/yaml/v3 v3.0.4 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/json-iterator/go v1.1.12 go: downloading gomodules.xyz/jsonpatch/v2 v2.5.0 go: downloading golang.org/x/time v0.14.0 go: downloading github.com/fxamacker/cbor/v2 v2.9.0 go: downloading golang.org/x/oauth2 v0.34.0 go: downloading golang.org/x/sys v0.39.0 go: downloading github.com/moby/spdystream v0.5.0 go: downloading sigs.k8s.io/scheduler-plugins v0.33.5 go: downloading volcano.sh/apis v1.13.0 go: downloading github.com/google/uuid v1.6.0 go: downloading k8s.io/apiextensions-apiserver v0.34.3 go: downloading github.com/google/btree v1.1.3 go: downloading golang.org/x/sync v0.19.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee go: downloading golang.org/x/text v0.32.0 go: downloading github.com/go-openapi/swag v0.25.4 go: downloading github.com/go-openapi/jsonreference v0.21.4 go: downloading github.com/x448/float16 v0.8.4 go: downloading github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f go: downloading golang.org/x/tools v0.40.0 go: downloading github.com/blang/semver/v4 v4.0.0 go: downloading go.opentelemetry.io/otel/trace v1.41.0 go: downloading github.com/emicklei/go-restful/v3 v3.13.0 go: downloading go.opentelemetry.io/otel v1.41.0 go: downloading github.com/go-openapi/jsonpointer v0.22.4 go: downloading github.com/go-openapi/swag/cmdutils v0.25.4 go: downloading github.com/go-openapi/swag/conv v0.25.4 go: downloading github.com/go-openapi/swag/fileutils v0.25.4 go: downloading github.com/go-openapi/swag/jsonname v0.25.4 go: downloading github.com/go-openapi/swag/jsonutils v0.25.4 go: downloading github.com/go-openapi/swag/loading v0.25.4 go: downloading github.com/go-openapi/swag/mangling v0.25.4 go: downloading github.com/go-openapi/swag/netutils v0.25.4 go: downloading github.com/go-openapi/swag/stringutils v0.25.4 go: downloading github.com/go-openapi/swag/typeutils v0.25.4 go: downloading github.com/go-openapi/swag/yamlutils v0.25.4 go: downloading gopkg.in/evanphx/json-patch.v4 v4.13.0 go: downloading golang.org/x/mod v0.31.0 === RUN TestRayJobWithClusterSelector rayjob_cluster_selector_test.go:27: [2026-09-17T14:02:42Z] Created ConfigMap test-ns-kdt9d/jobs successfully rayjob_cluster_selector_test.go:35: [2026-09-17T14:02:42Z] Created RayCluster test-ns-kdt9d/raycluster successfully rayjob_cluster_selector_test.go:37: [2026-09-17T14:02:42Z] Waiting for RayCluster test-ns-kdt9d/raycluster to become ready === RUN TestRayJobWithClusterSelector/Successful_RayJob === PAUSE TestRayJobWithClusterSelector/Successful_RayJob === RUN TestRayJobWithClusterSelector/Failing_RayJob === PAUSE TestRayJobWithClusterSelector/Failing_RayJob === RUN TestRayJobWithClusterSelector/RayJob_should_be_created_but_not_to_be_updated_when_managed_externally rayjob_cluster_selector_test.go:109: [2026-09-17T14:07:32Z] Created RayJob test-ns-kdt9d/managed-externally successfully --- PASS: TestRayJobWithClusterSelector/RayJob_should_be_created_but_not_to_be_updated_when_managed_externally (3.01s) === RUN TestRayJobWithClusterSelector/RayJob_should_not_be_created_due_to_managedBy_invalid_value --- PASS: TestRayJobWithClusterSelector/RayJob_should_not_be_created_due_to_managedBy_invalid_value (0.01s) === CONT TestRayJobWithClusterSelector/Successful_RayJob rayjob_cluster_selector_test.go:57: [2026-09-17T14:07:35Z] Created RayJob test-ns-kdt9d/counter successfully rayjob_cluster_selector_test.go:59: [2026-09-17T14:07:35Z] Waiting for RayJob test-ns-kdt9d/counter to complete --- PASS: TestRayJobWithClusterSelector/Successful_RayJob (13.09s) === CONT TestRayJobWithClusterSelector/Failing_RayJob rayjob_cluster_selector_test.go:81: [2026-09-17T14:07:48Z] Created RayJob test-ns-kdt9d/fail successfully rayjob_cluster_selector_test.go:83: [2026-09-17T14:07:48Z] Waiting for RayJob test-ns-kdt9d/fail to complete --- PASS: TestRayJobWithClusterSelector/Failing_RayJob (9.07s) test.go:114: [2026-09-17T14:07:57Z] Retrieving Pod Container test-ns-kdt9d/counter-9ttm6/ray-job-submitter logs test.go:102: [2026-09-17T14:07:57Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-09-17T14:07:57Z] Output directory has been created at: /tmp/TestRayJobWithClusterSelector3461153144/001 test.go:114: [2026-09-17T14:07:57Z] Retrieving Pod Container test-ns-kdt9d/fail-xb44l/ray-job-submitter logs test.go:114: [2026-09-17T14:07:57Z] Retrieving Pod Container test-ns-kdt9d/raycluster-head-b427t/ray-head logs test.go:114: [2026-09-17T14:07:57Z] Retrieving Pod Container test-ns-kdt9d/raycluster-head-b427t/kube-rbac-proxy logs test.go:114: [2026-09-17T14:07:57Z] Retrieving Pod Container test-ns-kdt9d/raycluster-small-group-worker-ksrx6/ray-worker logs --- PASS: TestRayJobWithClusterSelector (293.31s) === RUN TestRayJobLightWeightMode rayjob_lightweight_test.go:28: [2026-09-17T14:07:57Z] Created ConfigMap test-ns-27g8d/jobs successfully === RUN TestRayJobLightWeightMode/Successful_RayJob rayjob_lightweight_test.go:57: [2026-09-17T14:07:57Z] Created RayJob test-ns-27g8d/counter successfully rayjob_lightweight_test.go:59: [2026-09-17T14:07:57Z] Waiting for RayJob test-ns-27g8d/counter to complete --- PASS: TestRayJobLightWeightMode/Successful_RayJob (107.85s) === RUN TestRayJobLightWeightMode/Failing_RayJob_without_cluster_shutdown_after_finished rayjob_lightweight_test.go:91: [2026-09-17T14:09:45Z] Created RayJob test-ns-27g8d/fail successfully rayjob_lightweight_test.go:93: [2026-09-17T14:09:45Z] Waiting for RayJob test-ns-27g8d/fail to complete --- PASS: TestRayJobLightWeightMode/Failing_RayJob_without_cluster_shutdown_after_finished (36.30s) === RUN TestRayJobLightWeightMode/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. rayjob_lightweight_test.go:122: [2026-09-17T14:10:22Z] Created RayJob test-ns-27g8d/stop successfully rayjob_lightweight_test.go:124: [2026-09-17T14:10:22Z] Waiting for RayJob test-ns-27g8d/stop to be 'Running' rayjob_lightweight_test.go:128: [2026-09-17T14:12:09Z] Waiting for RayJob test-ns-27g8d/stop to be 'Complete' rayjob_lightweight_test.go:138: [2026-09-17T14:12:34Z] Deleted RayJob test-ns-27g8d/stop successfully --- PASS: TestRayJobLightWeightMode/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. (132.89s) === RUN TestRayJobLightWeightMode/Successful_RayJob_in_HTTP_mode_with_auth_token rayjob_lightweight_test.go:157: [2026-09-17T14:12:35Z] Created RayJob test-ns-27g8d/counter-auth successfully with auth token rayjob_lightweight_test.go:160: [2026-09-17T14:12:35Z] Waiting for RayCluster to be created rayjob_lightweight_test.go:170: [2026-09-17T14:12:36Z] Waiting for RayCluster test-ns-27g8d/counter-auth-ln7jw to become ready rayjob_lightweight_test.go:193: [2026-09-17T14:14:35Z] Waiting for RayJob test-ns-27g8d/counter-auth to complete rayjob_lightweight_test.go:205: [2026-09-17T14:14:41Z] RayJob test-ns-27g8d/counter-auth completed successfully with auth token --- PASS: TestRayJobLightWeightMode/Successful_RayJob_in_HTTP_mode_with_auth_token (126.89s) test.go:114: [2026-09-17T14:14:41Z] Retrieving Pod Container test-ns-27g8d/counter-auth-ln7jw-head-nq9x6/ray-head logs test.go:102: [2026-09-17T14:14:41Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-09-17T14:14:41Z] Output directory has been created at: /tmp/TestRayJobLightWeightMode846248939/001 test.go:114: [2026-09-17T14:14:41Z] Retrieving Pod Container test-ns-27g8d/counter-auth-ln7jw-head-nq9x6/kube-rbac-proxy logs test.go:114: [2026-09-17T14:14:41Z] Retrieving Pod Container test-ns-27g8d/counter-auth-ln7jw-small-group-worker-c64sr/ray-worker logs test.go:114: [2026-09-17T14:14:41Z] Retrieving Pod Container test-ns-27g8d/fail-glxgn-head-qxkj9/ray-head logs test.go:114: [2026-09-17T14:14:41Z] Retrieving Pod Container test-ns-27g8d/fail-glxgn-head-qxkj9/kube-rbac-proxy logs test.go:114: [2026-09-17T14:14:41Z] Retrieving Pod Container test-ns-27g8d/fail-glxgn-small-group-worker-qbg4b/ray-worker logs --- PASS: TestRayJobLightWeightMode (404.06s) === RUN TestRayJobSuspend rayjob_suspend_test.go:27: [2026-09-17T14:14:42Z] Created ConfigMap test-ns-gvpnr/jobs successfully === RUN TestRayJobSuspend/Suspend_the_RayJob_when_its_status_is_'Running',_and_then_resume_it. rayjob_suspend_test.go:41: [2026-09-17T14:14:42Z] Created RayJob test-ns-gvpnr/long-running successfully rayjob_suspend_test.go:43: [2026-09-17T14:14:42Z] Waiting for RayJob test-ns-gvpnr/long-running to be 'Running' rayjob_suspend_test.go:47: [2026-09-17T14:16:23Z] Suspend the RayJob test-ns-gvpnr/long-running rayjob_suspend_test.go:52: [2026-09-17T14:16:23Z] Waiting for RayJob test-ns-gvpnr/long-running to be 'Suspended' rayjob_suspend_test.go:68: [2026-09-17T14:16:24Z] Resume the RayJob by updating `suspend` to false. rayjob_suspend_test.go:78: [2026-09-17T14:18:07Z] Deleted RayJob test-ns-gvpnr/long-running successfully --- PASS: TestRayJobSuspend/Suspend_the_RayJob_when_its_status_is_'Running',_and_then_resume_it. (205.36s) === RUN TestRayJobSuspend/Create_a_suspended_RayJob,_and_then_resume_it. rayjob_suspend_test.go:97: [2026-09-17T14:18:07Z] Created RayJob test-ns-gvpnr/counter successfully rayjob_suspend_test.go:99: [2026-09-17T14:18:07Z] Waiting for RayJob test-ns-gvpnr/counter to be 'Suspended' rayjob_suspend_test.go:103: [2026-09-17T14:18:08Z] Resume the RayJob by updating `suspend` to false. rayjob_suspend_test.go:108: [2026-09-17T14:18:08Z] Waiting for RayJob test-ns-gvpnr/counter to complete rayjob_suspend_test.go:123: [2026-09-17T14:18:59Z] Deleted RayJob test-ns-gvpnr/counter successfully --- PASS: TestRayJobSuspend/Create_a_suspended_RayJob,_and_then_resume_it. (54.37s) test.go:102: [2026-09-17T14:19:01Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-09-17T14:19:01Z] Output directory has been created at: /tmp/TestRayJobSuspend727200210/001 --- PASS: TestRayJobSuspend (259.79s) === RUN TestRayJob rayjob_test.go:34: [2026-09-17T14:19:01Z] Created ConfigMap test-ns-bvqjw/jobs successfully === RUN TestRayJob/Successful_RayJob rayjob_test.go:51: [2026-09-17T14:19:01Z] Created RayJob test-ns-bvqjw/counter successfully rayjob_test.go:61: [2026-09-17T14:20:48Z] Waiting for RayJob test-ns-bvqjw/counter to complete rayjob_test.go:77: [2026-09-17T14:21:08Z] Checking that the RayJob status info has been set correctly. rayjob_test.go:94: [2026-09-17T14:21:08Z] Update `suspend` to true. However, since the RayJob is completed, the status should not be updated to `Suspended`. rayjob_test.go:104: [2026-09-17T14:21:38Z] Deleted RayJob test-ns-bvqjw/counter successfully --- PASS: TestRayJob/Successful_RayJob (156.60s) === RUN TestRayJob/Failing_RayJob_without_cluster_shutdown_after_finished rayjob_test.go:118: [2026-09-17T14:21:38Z] Created RayJob test-ns-bvqjw/fail successfully rayjob_test.go:120: [2026-09-17T14:21:38Z] Waiting for RayJob test-ns-bvqjw/fail to complete rayjob_test.go:143: [2026-09-17T14:23:30Z] Deleted RayJob test-ns-bvqjw/fail successfully --- PASS: TestRayJob/Failing_RayJob_without_cluster_shutdown_after_finished (112.69s) === RUN TestRayJob/Failing_submitter_K8s_Job rayjob_test.go:171: [2026-09-17T14:23:31Z] Created RayJob test-ns-bvqjw/fail-k8s-job successfully rayjob_test.go:173: [2026-09-17T14:23:31Z] Waiting for RayJob test-ns-bvqjw/fail-k8s-job to complete rayjob_test.go:196: [2026-09-17T14:24:50Z] Deleted RayJob test-ns-bvqjw/fail-k8s-job successfully --- PASS: TestRayJob/Failing_submitter_K8s_Job (79.63s) === RUN TestRayJob/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. rayjob_test.go:210: [2026-09-17T14:24:50Z] Created RayJob test-ns-bvqjw/stop successfully rayjob_test.go:212: [2026-09-17T14:24:50Z] Waiting for RayJob test-ns-bvqjw/stop to be 'Running' rayjob_test.go:216: [2026-09-17T14:25:22Z] Waiting for RayJob test-ns-bvqjw/stop to be 'Complete' rayjob_test.go:226: [2026-09-17T14:25:59Z] Deleted RayJob test-ns-bvqjw/stop successfully --- PASS: TestRayJob/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. (68.49s) === RUN TestRayJob/RuntimeEnvYAML_is_not_a_valid_YAML_string rayjob_test.go:238: [2026-09-17T14:25:59Z] Created RayJob test-ns-bvqjw/invalid-yamlstr successfully --- PASS: TestRayJob/RuntimeEnvYAML_is_not_a_valid_YAML_string (1.04s) === RUN TestRayJob/RayJob_name_too_long_with_48_characters rayjob_test.go:253: [2026-09-17T14:26:00Z] Created RayJob test-ns-bvqjw/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa successfully --- PASS: TestRayJob/RayJob_name_too_long_with_48_characters (1.04s) === RUN TestRayJob/RayJob_has_passed_ActiveDeadlineSeconds rayjob_test.go:272: [2026-09-17T14:26:01Z] Created RayJob test-ns-bvqjw/long-running successfully rayjob_test.go:275: [2026-09-17T14:26:01Z] Waiting for RayJob test-ns-bvqjw/long-running to be 'Complete' --- PASS: TestRayJob/RayJob_has_passed_ActiveDeadlineSeconds (7.11s) === RUN TestRayJob/RayJob_controller_recreates_the_head_Pod_if_it_is_deleted_while_the_job_is_running rayjob_test.go:292: [2026-09-17T14:26:08Z] Created RayJob test-ns-bvqjw/delete-head-after-submit successfully rayjob_test.go:295: [2026-09-17T14:26:08Z] Waiting for RayJob test-ns-bvqjw/delete-head-after-submit to be 'Running' rayjob_test.go:308: [2026-09-17T14:28:00Z] Deleting head Pod test-ns-bvqjw/delete-head-after-submit-nmww4-head-hnd99 for RayCluster delete-head-after-submit-nmww4 rayjob_test.go:337: [2026-09-17T14:29:36Z] Deleted RayJob test-ns-bvqjw/delete-head-after-submit successfully --- PASS: TestRayJob/RayJob_controller_recreates_the_head_Pod_if_it_is_deleted_while_the_job_is_running (207.71s) === RUN TestRayJob/RayJob_should_be_created,_but_not_updated_when_managed_externally rayjob_test.go:356: [2026-09-17T14:29:36Z] Created RayJob test-ns-bvqjw/managed-externally successfully rayjob_test.go:382: [2026-09-17T14:29:36Z] Deleted RayJob test-ns-bvqjw/managed-externally successfully --- PASS: TestRayJob/RayJob_should_be_created,_but_not_updated_when_managed_externally (0.05s) === RUN TestRayJob/RayJob_has_exceed_SubmitterFinishedTimeout rayjob_test.go:395: [2026-09-17T14:29:36Z] Created RayJob test-ns-bvqjw/submitter-timeout successfully rayjob_test.go:398: [2026-09-17T14:29:36Z] Waiting for RayJob test-ns-bvqjw/submitter-timeout deployment status to be 'Running' rayjob_test.go:401: [2026-09-17T14:31:28Z] Waiting for Ray job test-ns-bvqjw/submitter-timeout to be actually running in Ray cluster rayjob_test.go:406: [2026-09-17T14:31:40Z] Waiting for submitter job to be created rayjob_test.go:411: [2026-09-17T14:31:40Z] Updating submitter job status to complete rayjob_test.go:440: [2026-09-17T14:31:40Z] Successfully marked submitter job as completed at 2026-09-17 14:31:40.11884577 +0000 UTC m=+1737.686200940 rayjob_test.go:446: [2026-09-17T14:31:40Z] Waiting for RayJob test-ns-bvqjw/submitter-timeout to exceed SubmitterFinishedTimeout --- PASS: TestRayJob/RayJob_has_exceed_SubmitterFinishedTimeout (157.36s) === RUN TestRayJob/RayCluster_status_update_propagates_to_RayJob rayjob_test.go:476: [2026-09-17T14:32:13Z] Created RayJob test-ns-bvqjw/cluster-status-update successfully rayjob_test.go:479: Timed out after 613.850s. The function passed to Eventually returned the following error: <*url.Error | 0xc00052ec60>: Get "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/apis/ray.io/v1/namespaces/test-ns-bvqjw/rayjobs/cluster-status-update": dial tcp 3.218.14.122:6443: connect: connection refused { Op: "Get", URL: "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/apis/ray.io/v1/namespaces/test-ns-bvqjw/rayjobs/cluster-status-update", Err: <*net.OpError | 0xc000449180>{ Op: "dial", Net: "tcp", Source: nil, Addr: <*net.TCPAddr | 0xc0005b6750>{ IP: [3, 218, 14, 122], Port: 6443, Zone: "", }, Err: <*os.SyscallError | 0xc000028c00>{ Syscall: "connect", Err: 0x6f, }, }, } At one point, however, the function did return successfully. Yet, Eventually failed because the matcher was not satisfied: Expected : to equal : RUNNING testing.go:1811: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestRayJob/RayCluster_status_update_propagates_to_RayJob (613.88s) test.go:114: Error Trace: /tmp/kuberay_src/ray-operator/test/support/core.go:39 /tmp/kuberay_src/ray-operator/test/support/test.go:114 /usr/lib/golang/src/testing/testing.go:1308 /usr/lib/golang/src/testing/testing.go:1572 /usr/lib/golang/src/testing/testing.go:1928 /usr/lib/golang/src/runtime/panic.go:615 /usr/lib/golang/src/testing/testing.go:2008 /tmp/kuberay_src/ray-operator/test/e2erayjob/rayjob_test.go:466 Error: Received unexpected error: Get "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/test-ns-bvqjw/pods": dial tcp 3.218.14.122:6443: connect: connection refused Test: TestRayJob --- FAIL: TestRayJob (1422.19s) FAIL FAIL github.com/ray-project/kuberay/ray-operator/test/e2erayjob 2401.539s === RUN TestRayClusterAutoscaler === RUN TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaling_enabled raycluster_autoscaler_test.go:24: Error Trace: /tmp/kuberay_src/ray-operator/test/support/namespace.go:27 /tmp/kuberay_src/ray-operator/test/support/test.go:112 /tmp/kuberay_src/ray-operator/test/e2eautoscaler/raycluster_autoscaler_test.go:24 Error: Received unexpected error: Post "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces": dial tcp 35.174.96.41:6443: connect: connection refused Test: TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaling_enabled --- FAIL: TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaling_enabled (2.06s) === RUN TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaler_v2_enabled raycluster_autoscaler_test.go:24: Error Trace: /tmp/kuberay_src/ray-operator/test/support/namespace.go:27 /tmp/kuberay_src/ray-operator/test/support/test.go:112 /tmp/kuberay_src/ray-operator/test/e2eautoscaler/raycluster_autoscaler_test.go:24 Error: Received unexpected error: Post "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces": dial tcp 34.203.108.67:6443: connect: connection refused Test: TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaler_v2_enabled --- FAIL: TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaler_v2_enabled (1.09s) --- FAIL: TestRayClusterAutoscaler (3.15s) FAIL FAIL github.com/ray-project/kuberay/ray-operator/test/e2eautoscaler 3.164s === RUN TestRayClusterAuthOptions raycluster_auth_test.go:28: Error Trace: /tmp/kuberay_src/ray-operator/test/support/namespace.go:27 /tmp/kuberay_src/ray-operator/test/support/test.go:112 /tmp/kuberay_src/ray-operator/test/e2e/raycluster_auth_test.go:28 Error: Received unexpected error: Post "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces": dial tcp 3.218.14.122:6443: connect: connection refused Test: TestRayClusterAuthOptions --- FAIL: TestRayClusterAuthOptions (30.01s) === RUN TestRayClusterRHOAIImages raycluster_rhoai_images_test.go:34: listing CSVs in redhat-ods-operator: Get "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/apis/operators.coreos.com/v1alpha1/namespaces/redhat-ods-operator/clusterserviceversions": dial tcp 3.218.14.122:6443: connect: connection refused --- FAIL: TestRayClusterRHOAIImages (0.03s) FAIL FAIL github.com/ray-project/kuberay/ray-operator/test/e2e 30.061s === Failed === FAIL: test/e2e TestRayClusterAuthOptions (30.01s) raycluster_auth_test.go:28: Error Trace: /tmp/kuberay_src/ray-operator/test/support/namespace.go:27 /tmp/kuberay_src/ray-operator/test/support/test.go:112 /tmp/kuberay_src/ray-operator/test/e2e/raycluster_auth_test.go:28 Error: Received unexpected error: Post "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces": dial tcp 3.218.14.122:6443: connect: connection refused Test: TestRayClusterAuthOptions === FAIL: test/e2e TestRayClusterRHOAIImages (0.03s) raycluster_rhoai_images_test.go:34: listing CSVs in redhat-ods-operator: Get "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/apis/operators.coreos.com/v1alpha1/namespaces/redhat-ods-operator/clusterserviceversions": dial tcp 3.218.14.122:6443: connect: connection refused === FAIL: test/e2eautoscaler TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaling_enabled (2.06s) raycluster_autoscaler_test.go:24: Error Trace: /tmp/kuberay_src/ray-operator/test/support/namespace.go:27 /tmp/kuberay_src/ray-operator/test/support/test.go:112 /tmp/kuberay_src/ray-operator/test/e2eautoscaler/raycluster_autoscaler_test.go:24 Error: Received unexpected error: Post "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces": dial tcp 35.174.96.41:6443: connect: connection refused Test: TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaling_enabled === FAIL: test/e2eautoscaler TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaler_v2_enabled (1.09s) raycluster_autoscaler_test.go:24: Error Trace: /tmp/kuberay_src/ray-operator/test/support/namespace.go:27 /tmp/kuberay_src/ray-operator/test/support/test.go:112 /tmp/kuberay_src/ray-operator/test/e2eautoscaler/raycluster_autoscaler_test.go:24 Error: Received unexpected error: Post "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces": dial tcp 34.203.108.67:6443: connect: connection refused Test: TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaler_v2_enabled === FAIL: test/e2eautoscaler TestRayClusterAutoscaler (3.15s) === FAIL: test/e2erayjob TestRayJob/RayCluster_status_update_propagates_to_RayJob (613.88s) testing.go:1811: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test === FAIL: test/e2erayjob TestRayJob (1422.19s) rayjob_test.go:34: [2026-09-17T14:19:01Z] Created ConfigMap test-ns-bvqjw/jobs successfully rayjob_test.go:51: [2026-09-17T14:19:01Z] Created RayJob test-ns-bvqjw/counter successfully rayjob_test.go:61: [2026-09-17T14:20:48Z] Waiting for RayJob test-ns-bvqjw/counter to complete rayjob_test.go:77: [2026-09-17T14:21:08Z] Checking that the RayJob status info has been set correctly. rayjob_test.go:94: [2026-09-17T14:21:08Z] Update `suspend` to true. However, since the RayJob is completed, the status should not be updated to `Suspended`. rayjob_test.go:104: [2026-09-17T14:21:38Z] Deleted RayJob test-ns-bvqjw/counter successfully rayjob_test.go:118: [2026-09-17T14:21:38Z] Created RayJob test-ns-bvqjw/fail successfully rayjob_test.go:120: [2026-09-17T14:21:38Z] Waiting for RayJob test-ns-bvqjw/fail to complete rayjob_test.go:143: [2026-09-17T14:23:30Z] Deleted RayJob test-ns-bvqjw/fail successfully rayjob_test.go:171: [2026-09-17T14:23:31Z] Created RayJob test-ns-bvqjw/fail-k8s-job successfully rayjob_test.go:173: [2026-09-17T14:23:31Z] Waiting for RayJob test-ns-bvqjw/fail-k8s-job to complete rayjob_test.go:196: [2026-09-17T14:24:50Z] Deleted RayJob test-ns-bvqjw/fail-k8s-job successfully rayjob_test.go:210: [2026-09-17T14:24:50Z] Created RayJob test-ns-bvqjw/stop successfully rayjob_test.go:212: [2026-09-17T14:24:50Z] Waiting for RayJob test-ns-bvqjw/stop to be 'Running' rayjob_test.go:216: [2026-09-17T14:25:22Z] Waiting for RayJob test-ns-bvqjw/stop to be 'Complete' rayjob_test.go:226: [2026-09-17T14:25:59Z] Deleted RayJob test-ns-bvqjw/stop successfully rayjob_test.go:238: [2026-09-17T14:25:59Z] Created RayJob test-ns-bvqjw/invalid-yamlstr successfully rayjob_test.go:253: [2026-09-17T14:26:00Z] Created RayJob test-ns-bvqjw/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa successfully rayjob_test.go:272: [2026-09-17T14:26:01Z] Created RayJob test-ns-bvqjw/long-running successfully rayjob_test.go:275: [2026-09-17T14:26:01Z] Waiting for RayJob test-ns-bvqjw/long-running to be 'Complete' rayjob_test.go:292: [2026-09-17T14:26:08Z] Created RayJob test-ns-bvqjw/delete-head-after-submit successfully rayjob_test.go:295: [2026-09-17T14:26:08Z] Waiting for RayJob test-ns-bvqjw/delete-head-after-submit to be 'Running' rayjob_test.go:308: [2026-09-17T14:28:00Z] Deleting head Pod test-ns-bvqjw/delete-head-after-submit-nmww4-head-hnd99 for RayCluster delete-head-after-submit-nmww4 rayjob_test.go:337: [2026-09-17T14:29:36Z] Deleted RayJob test-ns-bvqjw/delete-head-after-submit successfully rayjob_test.go:356: [2026-09-17T14:29:36Z] Created RayJob test-ns-bvqjw/managed-externally successfully rayjob_test.go:382: [2026-09-17T14:29:36Z] Deleted RayJob test-ns-bvqjw/managed-externally successfully rayjob_test.go:395: [2026-09-17T14:29:36Z] Created RayJob test-ns-bvqjw/submitter-timeout successfully rayjob_test.go:398: [2026-09-17T14:29:36Z] Waiting for RayJob test-ns-bvqjw/submitter-timeout deployment status to be 'Running' rayjob_test.go:401: [2026-09-17T14:31:28Z] Waiting for Ray job test-ns-bvqjw/submitter-timeout to be actually running in Ray cluster rayjob_test.go:406: [2026-09-17T14:31:40Z] Waiting for submitter job to be created rayjob_test.go:411: [2026-09-17T14:31:40Z] Updating submitter job status to complete rayjob_test.go:440: [2026-09-17T14:31:40Z] Successfully marked submitter job as completed at 2026-09-17 14:31:40.11884577 +0000 UTC m=+1737.686200940 rayjob_test.go:446: [2026-09-17T14:31:40Z] Waiting for RayJob test-ns-bvqjw/submitter-timeout to exceed SubmitterFinishedTimeout rayjob_test.go:476: [2026-09-17T14:32:13Z] Created RayJob test-ns-bvqjw/cluster-status-update successfully rayjob_test.go:479: Timed out after 613.850s. The function passed to Eventually returned the following error: <*url.Error | 0xc00052ec60>: Get "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/apis/ray.io/v1/namespaces/test-ns-bvqjw/rayjobs/cluster-status-update": dial tcp 3.218.14.122:6443: connect: connection refused { Op: "Get", URL: "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/apis/ray.io/v1/namespaces/test-ns-bvqjw/rayjobs/cluster-status-update", Err: <*net.OpError | 0xc000449180>{ Op: "dial", Net: "tcp", Source: nil, Addr: <*net.TCPAddr | 0xc0005b6750>{ IP: [3, 218, 14, 122], Port: 6443, Zone: "", }, Err: <*os.SyscallError | 0xc000028c00>{ Syscall: "connect", Err: 0x6f, }, }, } At one point, however, the function did return successfully. Yet, Eventually failed because the matcher was not satisfied: Expected : to equal : RUNNING test.go:114: Error Trace: /tmp/kuberay_src/ray-operator/test/support/core.go:39 /tmp/kuberay_src/ray-operator/test/support/test.go:114 /usr/lib/golang/src/testing/testing.go:1308 /usr/lib/golang/src/testing/testing.go:1572 /usr/lib/golang/src/testing/testing.go:1928 /usr/lib/golang/src/runtime/panic.go:615 /usr/lib/golang/src/testing/testing.go:2008 /tmp/kuberay_src/ray-operator/test/e2erayjob/rayjob_test.go:466 Error: Received unexpected error: Get "https://a5b5c1aed2cc740199d1cdb0b50379a0-c801777cee8e3121.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/test-ns-bvqjw/pods": dial tcp 3.218.14.122:6443: connect: connection refused Test: TestRayJob DONE 30 tests, 7 failures in 2438.155s