Adding Tier1 kuberay e2e tests (including autoscaler, packages: ./test/e2erayjob ./test/e2eautoscaler) Adding Smoke kuberay e2e tests (packages: ./test/e2e) Selected tiers: Tier1 Smoke Test packages: ./test/e2erayjob ./test/e2eautoscaler ./test/e2e Running e2e tests matching: ^(TestRayJobWithClusterSelector|TestRayJob|TestRayJobSuspend|TestRayJobLightWeightMode|TestRayClusterAutoscaler|TestRayClusterAuthenticationRhoai|TestRayClusterAuthOptions)$ Test timeout: 60m go: downloading github.com/onsi/gomega v1.38.3 go: downloading github.com/stretchr/testify v1.11.1 go: downloading k8s.io/api v0.35.0 go: downloading k8s.io/apimachinery v0.35.0 go: downloading k8s.io/client-go v0.35.0 go: downloading k8s.io/utils v0.0.0-20251002143259-bc988d571ff4 go: downloading sigs.k8s.io/controller-runtime v0.22.4 go: downloading github.com/openshift/api v0.0.0-20251214014457-bfa868a22401 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading sigs.k8s.io/gateway-api v1.4.1 go: downloading github.com/onsi/ginkgo/v2 v2.27.3 go: downloading github.com/prometheus/client_golang v1.23.2 go: downloading github.com/go-logr/logr v1.4.3 go: downloading k8s.io/apiserver v0.34.3 go: downloading k8s.io/component-base v0.34.3 go: downloading github.com/Masterminds/semver/v3 v3.4.0 go: downloading sigs.k8s.io/yaml v1.6.0 go: downloading github.com/google/go-cmp v0.7.0 go: downloading sigs.k8s.io/randfill v1.0.0 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/cert-manager/cert-manager v1.18.2 go: downloading github.com/orcaman/concurrent-map/v2 v2.0.1 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading k8s.io/klog/v2 v2.130.1 go: downloading k8s.io/kube-openapi v0.0.0-20251125145642-4e65d59e963e go: downloading sigs.k8s.io/structured-merge-diff/v6 v6.3.1 go: downloading github.com/hashicorp/golang-lru/v2 v2.0.7 go: downloading github.com/smallnest/chanx v1.2.0 go: downloading sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 go: downloading github.com/evanphx/json-patch/v5 v5.9.11 go: downloading github.com/google/gnostic-models v0.7.1 go: downloading google.golang.org/protobuf v1.36.11 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/client_model v0.6.2 go: downloading github.com/prometheus/common v0.67.4 go: downloading github.com/prometheus/procfs v0.19.2 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading golang.org/x/net v0.48.0 go: downloading github.com/spf13/pflag v1.0.10 go: downloading golang.org/x/term v0.38.0 go: downloading github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 go: downloading go.yaml.in/yaml/v2 v2.4.3 go: downloading go.yaml.in/yaml/v3 v3.0.4 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/json-iterator/go v1.1.12 go: downloading gomodules.xyz/jsonpatch/v2 v2.5.0 go: downloading golang.org/x/sys v0.39.0 go: downloading golang.org/x/time v0.14.0 go: downloading github.com/fxamacker/cbor/v2 v2.9.0 go: downloading golang.org/x/oauth2 v0.34.0 go: downloading github.com/moby/spdystream v0.5.0 go: downloading sigs.k8s.io/scheduler-plugins v0.33.5 go: downloading volcano.sh/apis v1.13.0 go: downloading github.com/google/uuid v1.6.0 go: downloading k8s.io/apiextensions-apiserver v0.34.3 go: downloading github.com/google/btree v1.1.3 go: downloading golang.org/x/sync v0.19.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee go: downloading github.com/go-openapi/jsonreference v0.21.4 go: downloading github.com/go-openapi/swag v0.25.4 go: downloading golang.org/x/text v0.32.0 go: downloading github.com/x448/float16 v0.8.4 go: downloading github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f go: downloading golang.org/x/tools v0.40.0 go: downloading github.com/blang/semver/v4 v4.0.0 go: downloading go.opentelemetry.io/otel/trace v1.39.0 go: downloading github.com/emicklei/go-restful/v3 v3.13.0 go: downloading github.com/go-openapi/jsonpointer v0.22.4 go: downloading go.opentelemetry.io/otel v1.39.0 go: downloading github.com/go-openapi/swag/cmdutils v0.25.4 go: downloading github.com/go-openapi/swag/conv v0.25.4 go: downloading github.com/go-openapi/swag/fileutils v0.25.4 go: downloading github.com/go-openapi/swag/jsonname v0.25.4 go: downloading github.com/go-openapi/swag/jsonutils v0.25.4 go: downloading github.com/go-openapi/swag/loading v0.25.4 go: downloading github.com/go-openapi/swag/mangling v0.25.4 go: downloading github.com/go-openapi/swag/netutils v0.25.4 go: downloading github.com/go-openapi/swag/stringutils v0.25.4 go: downloading github.com/go-openapi/swag/typeutils v0.25.4 go: downloading github.com/go-openapi/swag/yamlutils v0.25.4 go: downloading gopkg.in/evanphx/json-patch.v4 v4.13.0 go: downloading golang.org/x/mod v0.31.0 === RUN TestRayJobWithClusterSelector rayjob_cluster_selector_test.go:27: [2026-07-21T08:47:51Z] Created ConfigMap test-ns-d6kwv/jobs successfully rayjob_cluster_selector_test.go:35: [2026-07-21T08:47:51Z] Created RayCluster test-ns-d6kwv/raycluster successfully rayjob_cluster_selector_test.go:37: [2026-07-21T08:47:51Z] Waiting for RayCluster test-ns-d6kwv/raycluster to become ready === RUN TestRayJobWithClusterSelector/Successful_RayJob === PAUSE TestRayJobWithClusterSelector/Successful_RayJob === RUN TestRayJobWithClusterSelector/Failing_RayJob === PAUSE TestRayJobWithClusterSelector/Failing_RayJob === RUN TestRayJobWithClusterSelector/RayJob_should_be_created_but_not_to_be_updated_when_managed_externally rayjob_cluster_selector_test.go:109: [2026-07-21T08:52:45Z] Created RayJob test-ns-d6kwv/managed-externally successfully --- PASS: TestRayJobWithClusterSelector/RayJob_should_be_created_but_not_to_be_updated_when_managed_externally (3.01s) === RUN TestRayJobWithClusterSelector/RayJob_should_not_be_created_due_to_managedBy_invalid_value --- PASS: TestRayJobWithClusterSelector/RayJob_should_not_be_created_due_to_managedBy_invalid_value (0.01s) === CONT TestRayJobWithClusterSelector/Successful_RayJob rayjob_cluster_selector_test.go:57: [2026-07-21T08:52:48Z] Created RayJob test-ns-d6kwv/counter successfully rayjob_cluster_selector_test.go:59: [2026-07-21T08:52:48Z] Waiting for RayJob test-ns-d6kwv/counter to complete --- PASS: TestRayJobWithClusterSelector/Successful_RayJob (15.11s) === CONT TestRayJobWithClusterSelector/Failing_RayJob rayjob_cluster_selector_test.go:81: [2026-07-21T08:53:03Z] Created RayJob test-ns-d6kwv/fail successfully rayjob_cluster_selector_test.go:83: [2026-07-21T08:53:03Z] Waiting for RayJob test-ns-d6kwv/fail to complete --- PASS: TestRayJobWithClusterSelector/Failing_RayJob (12.08s) test.go:114: [2026-07-21T08:53:15Z] Retrieving Pod Container test-ns-d6kwv/counter-tfhkw/ray-job-submitter logs test.go:102: [2026-07-21T08:53:15Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-21T08:53:15Z] Output directory has been created at: /tmp/TestRayJobWithClusterSelector1184962324/001 test.go:114: [2026-07-21T08:53:15Z] Retrieving Pod Container test-ns-d6kwv/fail-8h74g/ray-job-submitter logs test.go:114: [2026-07-21T08:53:15Z] Retrieving Pod Container test-ns-d6kwv/raycluster-head-kz9ls/ray-head logs test.go:114: [2026-07-21T08:53:15Z] Retrieving Pod Container test-ns-d6kwv/raycluster-head-kz9ls/kube-rbac-proxy logs test.go:114: [2026-07-21T08:53:15Z] Retrieving Pod Container test-ns-d6kwv/raycluster-small-group-worker-v979l/ray-worker logs --- PASS: TestRayJobWithClusterSelector (296.89s) === RUN TestRayJobLightWeightMode rayjob_lightweight_test.go:28: [2026-07-21T08:53:15Z] Created ConfigMap test-ns-g7r9j/jobs successfully === RUN TestRayJobLightWeightMode/Successful_RayJob rayjob_lightweight_test.go:57: [2026-07-21T08:53:15Z] Created RayJob test-ns-g7r9j/counter successfully rayjob_lightweight_test.go:59: [2026-07-21T08:53:15Z] Waiting for RayJob test-ns-g7r9j/counter to complete --- PASS: TestRayJobLightWeightMode/Successful_RayJob (97.60s) === RUN TestRayJobLightWeightMode/Failing_RayJob_without_cluster_shutdown_after_finished rayjob_lightweight_test.go:91: [2026-07-21T08:54:52Z] Created RayJob test-ns-g7r9j/fail successfully rayjob_lightweight_test.go:93: [2026-07-21T08:54:52Z] Waiting for RayJob test-ns-g7r9j/fail to complete --- PASS: TestRayJobLightWeightMode/Failing_RayJob_without_cluster_shutdown_after_finished (102.63s) === RUN TestRayJobLightWeightMode/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. rayjob_lightweight_test.go:122: [2026-07-21T08:56:35Z] Created RayJob test-ns-g7r9j/stop successfully rayjob_lightweight_test.go:124: [2026-07-21T08:56:35Z] Waiting for RayJob test-ns-g7r9j/stop to be 'Running' rayjob_lightweight_test.go:128: [2026-07-21T08:58:22Z] Waiting for RayJob test-ns-g7r9j/stop to be 'Complete' rayjob_lightweight_test.go:138: [2026-07-21T08:58:45Z] Deleted RayJob test-ns-g7r9j/stop successfully --- PASS: TestRayJobLightWeightMode/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. (129.84s) === RUN TestRayJobLightWeightMode/Successful_RayJob_in_HTTP_mode_with_auth_token rayjob_lightweight_test.go:157: [2026-07-21T08:58:45Z] Created RayJob test-ns-g7r9j/counter-auth successfully with auth token rayjob_lightweight_test.go:160: [2026-07-21T08:58:45Z] Waiting for RayCluster to be created rayjob_lightweight_test.go:170: [2026-07-21T08:58:46Z] Waiting for RayCluster test-ns-g7r9j/counter-auth-vftj8 to become ready rayjob_lightweight_test.go:193: [2026-07-21T09:00:31Z] Waiting for RayJob test-ns-g7r9j/counter-auth to complete rayjob_lightweight_test.go:205: [2026-07-21T09:00:37Z] RayJob test-ns-g7r9j/counter-auth completed successfully with auth token --- PASS: TestRayJobLightWeightMode/Successful_RayJob_in_HTTP_mode_with_auth_token (111.77s) test.go:114: [2026-07-21T09:00:37Z] Retrieving Pod Container test-ns-g7r9j/counter-auth-vftj8-head-w4rb6/ray-head logs test.go:102: [2026-07-21T09:00:37Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-21T09:00:37Z] Output directory has been created at: /tmp/TestRayJobLightWeightMode1913701506/001 test.go:114: [2026-07-21T09:00:37Z] Retrieving Pod Container test-ns-g7r9j/counter-auth-vftj8-head-w4rb6/kube-rbac-proxy logs test.go:114: [2026-07-21T09:00:37Z] Retrieving Pod Container test-ns-g7r9j/counter-auth-vftj8-small-group-worker-qtjjt/ray-worker logs test.go:114: [2026-07-21T09:00:37Z] Retrieving Pod Container test-ns-g7r9j/fail-sg4xn-head-cv99d/ray-head logs test.go:114: [2026-07-21T09:00:37Z] Retrieving Pod Container test-ns-g7r9j/fail-sg4xn-head-cv99d/kube-rbac-proxy logs test.go:114: [2026-07-21T09:00:37Z] Retrieving Pod Container test-ns-g7r9j/fail-sg4xn-small-group-worker-qsr4z/ray-worker logs --- PASS: TestRayJobLightWeightMode (441.96s) === RUN TestRayJobSuspend rayjob_suspend_test.go:27: [2026-07-21T09:00:37Z] Created ConfigMap test-ns-f56s8/jobs successfully === RUN TestRayJobSuspend/Suspend_the_RayJob_when_its_status_is_'Running',_and_then_resume_it. rayjob_suspend_test.go:41: [2026-07-21T09:00:37Z] Created RayJob test-ns-f56s8/long-running successfully rayjob_suspend_test.go:43: [2026-07-21T09:00:37Z] Waiting for RayJob test-ns-f56s8/long-running to be 'Running' rayjob_suspend_test.go:47: [2026-07-21T09:01:08Z] Suspend the RayJob test-ns-f56s8/long-running rayjob_suspend_test.go:52: [2026-07-21T09:01:08Z] Waiting for RayJob test-ns-f56s8/long-running to be 'Suspended' rayjob_suspend_test.go:68: [2026-07-21T09:01:09Z] Resume the RayJob by updating `suspend` to false. rayjob_suspend_test.go:78: [2026-07-21T09:02:56Z] Deleted RayJob test-ns-f56s8/long-running successfully --- PASS: TestRayJobSuspend/Suspend_the_RayJob_when_its_status_is_'Running',_and_then_resume_it. (138.97s) === RUN TestRayJobSuspend/Create_a_suspended_RayJob,_and_then_resume_it. rayjob_suspend_test.go:97: [2026-07-21T09:02:56Z] Created RayJob test-ns-f56s8/counter successfully rayjob_suspend_test.go:99: [2026-07-21T09:02:56Z] Waiting for RayJob test-ns-f56s8/counter to be 'Suspended' rayjob_suspend_test.go:103: [2026-07-21T09:02:57Z] Resume the RayJob by updating `suspend` to false. rayjob_suspend_test.go:108: [2026-07-21T09:02:57Z] Waiting for RayJob test-ns-f56s8/counter to complete rayjob_suspend_test.go:123: [2026-07-21T09:05:01Z] Deleted RayJob test-ns-f56s8/counter successfully --- PASS: TestRayJobSuspend/Create_a_suspended_RayJob,_and_then_resume_it. (125.95s) test.go:102: [2026-07-21T09:05:02Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-21T09:05:02Z] Output directory has been created at: /tmp/TestRayJobSuspend2217007425/001 --- PASS: TestRayJobSuspend (264.96s) === RUN TestRayJob rayjob_test.go:34: [2026-07-21T09:05:02Z] Created ConfigMap test-ns-lvctf/jobs successfully === RUN TestRayJob/Successful_RayJob rayjob_test.go:51: [2026-07-21T09:05:02Z] Created RayJob test-ns-lvctf/counter successfully rayjob_test.go:59: Timed out after 120.000s. Expected success, but got an error: <*errors.StatusError | 0xc0007bdb80>: services "counter-head-svc" not found { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "services \"counter-head-svc\" not found", Reason: "NotFound", Details: { Name: "counter-head-svc", Group: "", Kind: "services", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 404, }, } testing.go:1811: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestRayJob/Successful_RayJob (120.03s) test.go:114: [2026-07-21T09:07:02Z] Retrieving Pod Container test-ns-lvctf/counter-phz74-head-xlzck/ray-head logs test.go:102: [2026-07-21T09:07:02Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-21T09:07:02Z] Output directory has been created at: /tmp/TestRayJob3331003914/001 test.go:114: [2026-07-21T09:07:02Z] Retrieving Pod Container test-ns-lvctf/counter-phz74-head-xlzck/kube-rbac-proxy logs test.go:114: [2026-07-21T09:07:02Z] Retrieving Pod Container test-ns-lvctf/counter-phz74-small-group-worker-v85mk/ray-worker logs --- FAIL: TestRayJob (120.18s) FAIL FAIL github.com/ray-project/kuberay/ray-operator/test/e2erayjob 1151.189s === RUN TestRayClusterAutoscaler === RUN TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaling_enabled raycluster_autoscaler_test.go:30: [2026-07-21T09:07:04Z] Created ConfigMap test-ns-nccfz/scripts successfully raycluster_autoscaler_test.go:50: [2026-07-21T09:07:04Z] Created RayCluster test-ns-nccfz/ray-cluster successfully raycluster_autoscaler_test.go:59: [2026-07-21T09:07:25Z] Found head pod test-ns-nccfz/ray-cluster-head-b9qsl core.go:88: [2026-07-21T09:07:25Z] Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor1] core.go:101: [2026-07-21T09:07:26Z] Command stdout: core.go:102: [2026-07-21T09:07:26Z] Command stderr: 2026-07-21 09:07:25,656 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-21 09:07:25,659 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.132.0.33:6379... 2026-07-21 09:07:25,678 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at http://10.132.0.33:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( core.go:88: [2026-07-21T09:07:30Z] Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor2] core.go:101: [2026-07-21T09:07:32Z] Command stdout: core.go:102: [2026-07-21T09:07:32Z] Command stderr: 2026-07-21 09:07:31,435 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-21 09:07:31,437 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.132.0.33:6379... 2026-07-21 09:07:31,456 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at http://10.132.0.33:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( core.go:88: [2026-07-21T09:07:36Z] Executing command: [python /home/ray/test_scripts/terminate_detached_actor.py actor1] core.go:101: [2026-07-21T09:07:38Z] Command stdout: core.go:102: [2026-07-21T09:07:38Z] Command stderr: 2026-07-21 09:07:37,217 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-21 09:07:37,221 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.132.0.33:6379... 2026-07-21 09:07:37,240 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at http://10.132.0.33:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( core.go:88: [2026-07-21T09:10:23Z] Executing command: [python /home/ray/test_scripts/terminate_detached_actor.py actor2] core.go:101: [2026-07-21T09:10:24Z] Command stdout: core.go:102: [2026-07-21T09:10:24Z] Command stderr: 2026-07-21 09:10:23,864 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-21 09:10:23,868 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.132.0.33:6379... 2026-07-21 09:10:23,887 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at http://10.132.0.33:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( test.go:114: [2026-07-21T09:11:30Z] Retrieving Pod Container test-ns-nccfz/ray-cluster-head-b9qsl/ray-head logs test.go:102: [2026-07-21T09:11:30Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-21T09:11:30Z] Output directory has been created at: /tmp/TestRayClusterAutoscalerCreate_a_RayCluster_with_autoscaling_en3536267114/001 test.go:114: [2026-07-21T09:11:30Z] Retrieving Pod Container test-ns-nccfz/ray-cluster-head-b9qsl/autoscaler logs test.go:114: [2026-07-21T09:11:30Z] Retrieving Pod Container test-ns-nccfz/ray-cluster-head-b9qsl/kube-rbac-proxy logs test.go:114: [2026-07-21T09:11:30Z] Retrieving Pod Container test-ns-nccfz/ray-cluster-small-group-worker-s9fcs/ray-worker logs --- PASS: TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaling_enabled (265.72s) === RUN TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaler_v2_enabled raycluster_autoscaler_test.go:30: [2026-07-21T09:11:30Z] Created ConfigMap test-ns-lsmn5/scripts successfully raycluster_autoscaler_test.go:50: [2026-07-21T09:11:30Z] Created RayCluster test-ns-lsmn5/ray-cluster successfully raycluster_autoscaler_test.go:59: [2026-07-21T09:13:06Z] Found head pod test-ns-lsmn5/ray-cluster-head-tgdsh core.go:88: [2026-07-21T09:13:06Z] Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor1] core.go:101: [2026-07-21T09:13:08Z] Command stdout: core.go:102: [2026-07-21T09:13:08Z] Command stderr: 2026-07-21 09:13:06,802 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-21 09:13:06,804 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.133.0.34:6379... 2026-07-21 09:13:06,822 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at http://10.133.0.34:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( core.go:88: [2026-07-21T09:13:10Z] Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor2] core.go:101: [2026-07-21T09:13:11Z] Command stdout: core.go:102: [2026-07-21T09:13:11Z] Command stderr: 2026-07-21 09:13:10,589 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-21 09:13:10,592 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.133.0.34:6379... 2026-07-21 09:13:10,610 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at http://10.133.0.34:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( core.go:88: [2026-07-21T09:13:15Z] Executing command: [python /home/ray/test_scripts/terminate_detached_actor.py actor1] core.go:101: [2026-07-21T09:13:17Z] Command stdout: core.go:102: [2026-07-21T09:13:17Z] Command stderr: 2026-07-21 09:13:16,295 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-21 09:13:16,299 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.133.0.34:6379... 2026-07-21 09:13:16,318 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at http://10.133.0.34:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( core.go:88: [2026-07-21T09:16:07Z] Executing command: [python /home/ray/test_scripts/terminate_detached_actor.py actor2] core.go:101: [2026-07-21T09:16:08Z] Command stdout: core.go:102: [2026-07-21T09:16:08Z] Command stderr: 2026-07-21 09:16:07,899 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-21 09:16:07,904 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.133.0.34:6379... 2026-07-21 09:16:07,923 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at http://10.133.0.34:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( test.go:114: [2026-07-21T09:17:13Z] Retrieving Pod Container test-ns-lsmn5/ray-cluster-head-tgdsh/ray-head logs test.go:102: [2026-07-21T09:17:13Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-21T09:17:13Z] Output directory has been created at: /tmp/TestRayClusterAutoscalerCreate_a_RayCluster_with_autoscaler_v2_739644737/001 test.go:114: [2026-07-21T09:17:13Z] Retrieving Pod Container test-ns-lsmn5/ray-cluster-head-tgdsh/autoscaler logs test.go:114: [2026-07-21T09:17:13Z] Retrieving Pod Container test-ns-lsmn5/ray-cluster-head-tgdsh/kube-rbac-proxy logs test.go:114: [2026-07-21T09:17:13Z] Retrieving Pod Container test-ns-lsmn5/ray-cluster-small-group-worker-qft9c/ray-worker logs --- PASS: TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaler_v2_enabled (342.86s) --- PASS: TestRayClusterAutoscaler (608.58s) PASS ok github.com/ray-project/kuberay/ray-operator/test/e2eautoscaler 608.600s === RUN TestRayClusterAuthOptions === RUN TestRayClusterAuthOptions/RayCluster_with_token_authentication_enabled === PAUSE TestRayClusterAuthOptions/RayCluster_with_token_authentication_enabled === CONT TestRayClusterAuthOptions/RayCluster_with_token_authentication_enabled raycluster_auth_test.go:38: [2026-07-21T09:17:15Z] Created RayCluster test-ns-sgzw5/raycluster-auth-token successfully with AuthModeToken raycluster_auth_test.go:40: [2026-07-21T09:17:15Z] Waiting for RayCluster test-ns-sgzw5/raycluster-auth-token to become ready === RUN TestRayClusterAuthOptions/Submit_job_with_auth_token_should_succeed raycluster_auth_test.go:65: [2026-07-21T09:17:49Z] Testing job submission WITH auth token core.go:88: [2026-07-21T09:17:49Z] Executing command: [bash -c RAY_AUTH_TOKEN=dCFmxDZxJqzYaVglO88HVvT2DGGX+lLPQTotdr827X0= ray job submit --address http://127.0.0.1:8265 --submission-id test-job-with-auth-1784625469 --no-wait -- python -c 'import ray; ray.init(); print("Job with auth succeeded")'] core.go:101: [2026-07-21T09:17:52Z] Command stdout: 2026-07-21 09:17:51,420 INFO cli.py:41 -- Job submission server address: http://10.132.0.36:8265 2026-07-21 09:17:52,254 SUCC cli.py:65 -- ---------------------------------------------------------- 2026-07-21 09:17:52,254 SUCC cli.py:66 -- Job 'test-job-with-auth-1784625469' submitted successfully 2026-07-21 09:17:52,254 SUCC cli.py:67 -- ---------------------------------------------------------- 2026-07-21 09:17:52,254 INFO cli.py:326 -- Next steps 2026-07-21 09:17:52,254 INFO cli.py:327 -- Query the logs of the job: 2026-07-21 09:17:52,254 INFO cli.py:329 -- ray job logs test-job-with-auth-1784625469 2026-07-21 09:17:52,254 INFO cli.py:331 -- Query the status of the job: 2026-07-21 09:17:52,254 INFO cli.py:333 -- ray job status test-job-with-auth-1784625469 2026-07-21 09:17:52,254 INFO cli.py:335 -- Request the job to be stopped: 2026-07-21 09:17:52,254 INFO cli.py:337 -- ray job stop test-job-with-auth-1784625469 core.go:102: [2026-07-21T09:17:52Z] Command stderr: core.go:88: [2026-07-21T09:17:52Z] Executing command: [bash -c RAY_AUTH_TOKEN=dCFmxDZxJqzYaVglO88HVvT2DGGX+lLPQTotdr827X0= ray job status --address http://127.0.0.1:8265 test-job-with-auth-1784625469] core.go:101: [2026-07-21T09:17:54Z] Command stdout: 2026-07-21 09:17:54,476 INFO cli.py:41 -- Job submission server address: http://10.132.0.36:8265 2026-07-21 09:17:54,480 INFO cli.py:91 -- Status for job 'test-job-with-auth-1784625469': RUNNING 2026-07-21 09:17:54,480 INFO cli.py:93 -- Status message: Job is currently running. core.go:102: [2026-07-21T09:17:54Z] Command stderr: core.go:88: [2026-07-21T09:17:55Z] Executing command: [bash -c RAY_AUTH_TOKEN=dCFmxDZxJqzYaVglO88HVvT2DGGX+lLPQTotdr827X0= ray job status --address http://127.0.0.1:8265 test-job-with-auth-1784625469] core.go:101: [2026-07-21T09:17:57Z] Command stdout: 2026-07-21 09:17:57,538 INFO cli.py:41 -- Job submission server address: http://10.132.0.36:8265 2026-07-21 09:17:57,541 SUCC cli.py:65 -- --------------------------------------------- 2026-07-21 09:17:57,541 SUCC cli.py:66 -- Job 'test-job-with-auth-1784625469' succeeded 2026-07-21 09:17:57,541 SUCC cli.py:67 -- --------------------------------------------- core.go:102: [2026-07-21T09:17:57Z] Command stderr: raycluster_auth_test.go:92: [2026-07-21T09:17:57Z] Successfully submitted and verified job with auth token --- PASS: TestRayClusterAuthOptions/Submit_job_with_auth_token_should_succeed (8.17s) === RUN TestRayClusterAuthOptions/Submit_job_with_incorrect_auth_token_should_fail raycluster_auth_test.go:96: [2026-07-21T09:17:57Z] Testing job submission WITH incorrect auth token (should fail) core.go:88: [2026-07-21T09:17:57Z] Executing command: [bash -c RAY_AUTH_TOKEN=incorrect-token-12345 ray job submit --address http://127.0.0.1:8265 --submission-id test-job-bad-auth-1784625477 --no-wait -- python -c 'print("Should not run")'] core.go:101: [2026-07-21T09:17:59Z] Command stdout: core.go:102: [2026-07-21T09:17:59Z] Command stderr: [2026-07-21 09:17:59,637 W 941 941] gcs_client.cc:205: Failed to get cluster ID from GCS server: Unauthenticated: RPC error: InvalidAuthToken: Authentication token is missing or incorrect Traceback (most recent call last): File "/opt/app-root/bin/ray", line 8, in sys.exit(main()) ^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/scripts/scripts.py", line 2892, in main return cli() ^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1514, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1435, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1902, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1902, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1298, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 853, in invoke return callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/dashboard/modules/job/cli_utils.py", line 54, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/autoscaler/_private/cli_logger.py", line 823, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/dashboard/modules/job/cli.py", line 303, in submit client = _get_sdk_client( ^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/dashboard/modules/job/cli.py", line 34, in _get_sdk_client client = JobSubmissionClient( ^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/dashboard/modules/job/sdk.py", line 101, in __init__ api_server_url = get_address_for_submission_client(address) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/dashboard/utils.py", line 733, in get_address_for_submission_client address = ray_address_to_api_server_url(address) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/dashboard/utils.py", line 676, in ray_address_to_api_server_url gcs_client = GcsClient(address=address) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "python/ray/_raylet.pyx", line 2770, in ray._raylet.GcsClient.__cinit__ File "python/ray/includes/gcs_client.pxi", line 73, in ray._raylet.InnerGcsClient.standalone File "python/ray/includes/common.pxi", line 143, in ray._raylet.check_status_timeout_as_rpc_error File "python/ray/includes/common.pxi", line 116, in ray._raylet.check_status ray.exceptions.AuthenticationError: RPC error: InvalidAuthToken: Authentication token is missing or incorrect. Ensure that the token for the cluster is available in a local file (e.g., ~/.ray/auth_token or via RAY_AUTH_TOKEN_PATH) or as the `RAY_AUTH_TOKEN` environment variable. To generate a token for local development, use `ray get-auth-token --generate` For remote clusters, ensure that the token is propagated to all nodes of the cluster when token authentication is enabled. For more information, see: https://docs.ray.io/en/latest/ray-security/token-auth.html raycluster_auth_test.go:113: [2026-07-21T09:17:59Z] Job submission correctly rejected with incorrect auth token --- PASS: TestRayClusterAuthOptions/Submit_job_with_incorrect_auth_token_should_fail (2.11s) --- PASS: TestRayClusterAuthOptions/RayCluster_with_token_authentication_enabled (44.58s) test.go:114: [2026-07-21T09:17:59Z] Retrieving Pod Container test-ns-sgzw5/raycluster-auth-token-head-dsqqc/ray-head logs test.go:102: [2026-07-21T09:17:59Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-21T09:17:59Z] Output directory has been created at: /tmp/TestRayClusterAuthOptions850427327/001 test.go:114: [2026-07-21T09:17:59Z] Retrieving Pod Container test-ns-sgzw5/raycluster-auth-token-head-dsqqc/kube-rbac-proxy logs test.go:114: [2026-07-21T09:17:59Z] Retrieving Pod Container test-ns-sgzw5/raycluster-auth-token-small-group-worker-2dwht/ray-worker logs --- PASS: TestRayClusterAuthOptions (0.11s) PASS ok github.com/ray-project/kuberay/ray-operator/test/e2e 44.709s === Failed === FAIL: test/e2erayjob TestRayJob/Successful_RayJob (120.03s) testing.go:1811: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test === FAIL: test/e2erayjob TestRayJob (120.18s) rayjob_test.go:34: [2026-07-21T09:05:02Z] Created ConfigMap test-ns-lvctf/jobs successfully rayjob_test.go:51: [2026-07-21T09:05:02Z] Created RayJob test-ns-lvctf/counter successfully rayjob_test.go:59: Timed out after 120.000s. Expected success, but got an error: <*errors.StatusError | 0xc0007bdb80>: services "counter-head-svc" not found { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "services \"counter-head-svc\" not found", Reason: "NotFound", Details: { Name: "counter-head-svc", Group: "", Kind: "services", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 404, }, } test.go:114: [2026-07-21T09:07:02Z] Retrieving Pod Container test-ns-lvctf/counter-phz74-head-xlzck/ray-head logs test.go:102: [2026-07-21T09:07:02Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-21T09:07:02Z] Output directory has been created at: /tmp/TestRayJob3331003914/001 test.go:114: [2026-07-21T09:07:02Z] Retrieving Pod Container test-ns-lvctf/counter-phz74-head-xlzck/kube-rbac-proxy logs test.go:114: [2026-07-21T09:07:02Z] Retrieving Pod Container test-ns-lvctf/counter-phz74-small-group-worker-v85mk/ray-worker logs DONE 22 tests, 2 failures in 1808.757s