Adding Tier1 kuberay e2e tests (including autoscaler) Adding Smoke kuberay e2e tests (authentication validation) Running e2e tests matching: ^(TestRayJobWithClusterSelector|TestRayJob|TestRayJobSuspend|TestRayJobLightWeightMode|TestRayClusterAutoscaler|TestRayClusterAuthentication)$ Running packages: ./test/e2e ./test/e2eautoscaler Test timeout: 60m go: downloading k8s.io/api v0.34.1 go: downloading github.com/stretchr/testify v1.11.0 go: downloading k8s.io/apimachinery v0.34.1 go: downloading k8s.io/client-go v0.34.1 go: downloading github.com/onsi/gomega v1.37.0 go: downloading k8s.io/utils v0.0.0-20250820121507-0af2bda4dd1d go: downloading github.com/openshift/api v0.0.0-20250602203052-b29811a290c7 go: downloading github.com/pkg/errors v0.9.1 go: downloading sigs.k8s.io/controller-runtime v0.22.1 go: downloading github.com/onsi/ginkgo/v2 v2.23.4 go: downloading github.com/google/go-cmp v0.7.0 go: downloading github.com/Masterminds/semver/v3 v3.3.1 go: downloading sigs.k8s.io/yaml v1.6.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading sigs.k8s.io/randfill v1.0.0 go: downloading github.com/go-logr/logr v1.4.3 go: downloading k8s.io/apiserver v0.34.1 go: downloading k8s.io/component-base v0.34.1 go: downloading k8s.io/klog/v2 v2.130.1 go: downloading sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 go: downloading k8s.io/kube-openapi v0.0.0-20250814151709-d7b6acb124c3 go: downloading sigs.k8s.io/structured-merge-diff/v6 v6.3.0 go: downloading github.com/google/gnostic-models v0.7.0 go: downloading google.golang.org/protobuf v1.36.8 go: downloading github.com/evanphx/json-patch/v5 v5.9.11 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading golang.org/x/net v0.43.0 go: downloading github.com/spf13/pflag v1.0.7 go: downloading golang.org/x/term v0.34.0 go: downloading github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading go.yaml.in/yaml/v2 v2.4.2 go: downloading go.yaml.in/yaml/v3 v3.0.4 go: downloading github.com/prometheus/client_golang v1.23.0 go: downloading gomodules.xyz/jsonpatch/v2 v2.4.0 go: downloading golang.org/x/time v0.12.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/fxamacker/cbor/v2 v2.9.0 go: downloading golang.org/x/oauth2 v0.30.0 go: downloading golang.org/x/sys v0.35.0 go: downloading github.com/moby/spdystream v0.5.0 go: downloading golang.org/x/text v0.28.0 go: downloading github.com/go-openapi/jsonreference v0.21.0 go: downloading github.com/go-openapi/swag v0.23.1 go: downloading github.com/google/uuid v1.6.0 go: downloading k8s.io/apiextensions-apiserver v0.34.1 go: downloading github.com/x448/float16 v0.8.4 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee go: downloading github.com/prometheus/client_model v0.6.2 go: downloading github.com/prometheus/common v0.65.0 go: downloading github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f go: downloading github.com/blang/semver/v4 v4.0.0 go: downloading github.com/prometheus/procfs v0.17.0 go: downloading go.opentelemetry.io/otel/trace v1.35.0 go: downloading go.opentelemetry.io/otel v1.35.0 go: downloading github.com/emicklei/go-restful/v3 v3.13.0 go: downloading github.com/go-openapi/jsonpointer v0.21.2 go: downloading github.com/google/btree v1.1.3 go: downloading golang.org/x/sync v0.16.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading gopkg.in/evanphx/json-patch.v4 v4.13.0 go: downloading github.com/mailru/easyjson v0.9.0 go: downloading github.com/josharian/intern v1.0.0 === RUN TestRayClusterAuthentication raycluster_auth_test.go:34: [2026-07-13T15:43:24Z] Created RayCluster test-ns-k9mkf/raycluster-auth with authentication enabled raycluster_auth_test.go:36: [2026-07-13T15:43:24Z] Waiting for RayCluster test-ns-k9mkf/raycluster-auth to become ready === RUN TestRayClusterAuthentication/Auth_proxy_sidecar_is_injected_into_head_pod --- PASS: TestRayClusterAuthentication/Auth_proxy_sidecar_is_injected_into_head_pod (0.00s) === RUN TestRayClusterAuthentication/Unauthenticated_request_to_auth_proxy_is_rejected core.go:87: [2026-07-13T15:49:04Z] Executing command: [python3 -c import urllib.request, ssl ctx = ssl.create_default_context() ctx.check_hostname = False ctx.verify_mode = ssl.CERT_NONE try: urllib.request.urlopen(urllib.request.Request('https://localhost:8443/'), context=ctx, timeout=5) print('STATUS:200') except urllib.error.HTTPError as e: print('STATUS:' + str(e.code)) except Exception as e: print('ERROR:' + str(e)) ] core.go:100: [2026-07-13T15:49:04Z] Command stdout: STATUS:401 core.go:101: [2026-07-13T15:49:04Z] Command stderr: --- PASS: TestRayClusterAuthentication/Unauthenticated_request_to_auth_proxy_is_rejected (0.16s) test.go:114: [2026-07-13T15:49:04Z] Retrieving Pod Container test-ns-k9mkf/raycluster-auth-head-czcms/ray-head logs test.go:102: [2026-07-13T15:49:04Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-13T15:49:04Z] Output directory has been created at: /tmp/TestRayClusterAuthentication915205856/001 test.go:114: [2026-07-13T15:49:04Z] Retrieving Pod Container test-ns-k9mkf/raycluster-auth-head-czcms/kube-rbac-proxy logs test.go:114: [2026-07-13T15:49:04Z] Retrieving Pod Container test-ns-k9mkf/raycluster-auth-small-group-worker-z4nbk/ray-worker logs --- PASS: TestRayClusterAuthentication (340.30s) === RUN TestRayJobWithClusterSelector rayjob_cluster_selector_test.go:27: [2026-07-13T15:49:04Z] Created ConfigMap test-ns-qbjjb/jobs successfully rayjob_cluster_selector_test.go:35: [2026-07-13T15:49:04Z] Created RayCluster test-ns-qbjjb/raycluster successfully rayjob_cluster_selector_test.go:37: [2026-07-13T15:49:04Z] Waiting for RayCluster test-ns-qbjjb/raycluster to become ready === RUN TestRayJobWithClusterSelector/Successful_RayJob === PAUSE TestRayJobWithClusterSelector/Successful_RayJob === RUN TestRayJobWithClusterSelector/Failing_RayJob === PAUSE TestRayJobWithClusterSelector/Failing_RayJob === RUN TestRayJobWithClusterSelector/RayJob_should_be_created_but_not_to_be_updated_when_managed_externally === PAUSE TestRayJobWithClusterSelector === RUN TestRayJobLightWeightMode rayjob_lightweight_test.go:27: [2026-07-13T15:49:36Z] Created ConfigMap test-ns-5xvz7/jobs successfully === RUN TestRayJobLightWeightMode/Successful_RayJob rayjob_lightweight_test.go:56: [2026-07-13T15:49:36Z] Created RayJob test-ns-5xvz7/counter successfully rayjob_lightweight_test.go:58: [2026-07-13T15:49:36Z] Waiting for RayJob test-ns-5xvz7/counter to complete --- PASS: TestRayJobLightWeightMode/Successful_RayJob (27.17s) === RUN TestRayJobLightWeightMode/Failing_RayJob_without_cluster_shutdown_after_finished rayjob_lightweight_test.go:90: [2026-07-13T15:50:03Z] Created RayJob test-ns-5xvz7/fail successfully rayjob_lightweight_test.go:92: [2026-07-13T15:50:03Z] Waiting for RayJob test-ns-5xvz7/fail to complete --- PASS: TestRayJobLightWeightMode/Failing_RayJob_without_cluster_shutdown_after_finished (96.50s) === RUN TestRayJobLightWeightMode/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. rayjob_lightweight_test.go:121: [2026-07-13T15:51:40Z] Created RayJob test-ns-5xvz7/stop successfully rayjob_lightweight_test.go:123: [2026-07-13T15:51:40Z] Waiting for RayJob test-ns-5xvz7/stop to be 'Running' rayjob_lightweight_test.go:127: [2026-07-13T15:53:27Z] Waiting for RayJob test-ns-5xvz7/stop to be 'Complete' rayjob_lightweight_test.go:137: [2026-07-13T15:53:53Z] Deleted RayJob test-ns-5xvz7/stop successfully --- PASS: TestRayJobLightWeightMode/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. (132.68s) test.go:114: [2026-07-13T15:53:53Z] Retrieving Pod Container test-ns-5xvz7/fail-pbfxx-head-27cb9/ray-head logs test.go:102: [2026-07-13T15:53:53Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-13T15:53:53Z] Output directory has been created at: /tmp/TestRayJobLightWeightMode3925777383/001 test.go:114: [2026-07-13T15:53:53Z] Retrieving Pod Container test-ns-5xvz7/fail-pbfxx-head-27cb9/kube-rbac-proxy logs test.go:114: [2026-07-13T15:53:53Z] Retrieving Pod Container test-ns-5xvz7/fail-pbfxx-small-group-worker-bhr2g/ray-worker logs test.go:114: [2026-07-13T15:53:53Z] Retrieving Pod Container test-ns-5xvz7/stop-bvlzx-head-cwv4c/ray-head logs test.go:114: [2026-07-13T15:53:53Z] Retrieving Pod Container test-ns-5xvz7/stop-bvlzx-head-cwv4c/kube-rbac-proxy logs test.go:114: [2026-07-13T15:53:53Z] Retrieving Pod Container test-ns-5xvz7/stop-bvlzx-small-group-worker-r2kxc/ray-worker logs --- PASS: TestRayJobLightWeightMode (256.47s) === RUN TestRayJobSuspend rayjob_suspend_test.go:27: [2026-07-13T15:53:53Z] Created ConfigMap test-ns-88gnn/jobs successfully === RUN TestRayJobSuspend/Suspend_the_RayJob_when_its_status_is_'Running',_and_then_resume_it. rayjob_suspend_test.go:41: [2026-07-13T15:53:53Z] Created RayJob test-ns-88gnn/long-running successfully rayjob_suspend_test.go:43: [2026-07-13T15:53:53Z] Waiting for RayJob test-ns-88gnn/long-running to be 'Running' rayjob_suspend_test.go:47: [2026-07-13T15:55:40Z] Suspend the RayJob test-ns-88gnn/long-running rayjob_suspend_test.go:52: [2026-07-13T15:55:40Z] Waiting for RayJob test-ns-88gnn/long-running to be 'Suspended' rayjob_suspend_test.go:68: [2026-07-13T15:55:41Z] Resume the RayJob by updating `suspend` to false. rayjob_suspend_test.go:78: [2026-07-13T15:57:19Z] Deleted RayJob test-ns-88gnn/long-running successfully --- PASS: TestRayJobSuspend/Suspend_the_RayJob_when_its_status_is_'Running',_and_then_resume_it. (206.09s) === RUN TestRayJobSuspend/Create_a_suspended_RayJob,_and_then_resume_it. rayjob_suspend_test.go:97: [2026-07-13T15:57:19Z] Created RayJob test-ns-88gnn/counter successfully rayjob_suspend_test.go:99: [2026-07-13T15:57:19Z] Waiting for RayJob test-ns-88gnn/counter to be 'Suspended' rayjob_suspend_test.go:103: [2026-07-13T15:57:20Z] Resume the RayJob by updating `suspend` to false. rayjob_suspend_test.go:108: [2026-07-13T15:57:20Z] Waiting for RayJob test-ns-88gnn/counter to complete rayjob_suspend_test.go:123: [2026-07-13T15:59:33Z] Deleted RayJob test-ns-88gnn/counter successfully --- PASS: TestRayJobSuspend/Create_a_suspended_RayJob,_and_then_resume_it. (135.77s) test.go:102: [2026-07-13T15:59:35Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-13T15:59:35Z] Output directory has been created at: /tmp/TestRayJobSuspend3423160432/001 --- PASS: TestRayJobSuspend (341.90s) === RUN TestRayJob rayjob_test.go:29: [2026-07-13T15:59:35Z] Created ConfigMap test-ns-x5qvg/jobs successfully === RUN TestRayJob/Successful_RayJob rayjob_test.go:46: [2026-07-13T15:59:35Z] Created RayJob test-ns-x5qvg/counter successfully rayjob_test.go:48: [2026-07-13T15:59:35Z] Waiting for RayJob test-ns-x5qvg/counter to complete rayjob_test.go:64: [2026-07-13T16:00:29Z] Checking that the RayJob status info has been set correctly. rayjob_test.go:81: [2026-07-13T16:00:29Z] Update `suspend` to true. However, since the RayJob is completed, the status should not be updated to `Suspended`. rayjob_test.go:91: [2026-07-13T16:00:59Z] Deleted RayJob test-ns-x5qvg/counter successfully --- PASS: TestRayJob/Successful_RayJob (84.37s) === RUN TestRayJob/Failing_RayJob_without_cluster_shutdown_after_finished rayjob_test.go:105: [2026-07-13T16:00:59Z] Created RayJob test-ns-x5qvg/fail successfully rayjob_test.go:107: [2026-07-13T16:00:59Z] Waiting for RayJob test-ns-x5qvg/fail to complete rayjob_test.go:130: [2026-07-13T16:03:06Z] Deleted RayJob test-ns-x5qvg/fail successfully --- PASS: TestRayJob/Failing_RayJob_without_cluster_shutdown_after_finished (127.76s) === RUN TestRayJob/Failing_submitter_K8s_Job rayjob_test.go:158: [2026-07-13T16:03:07Z] Created RayJob test-ns-x5qvg/fail-k8s-job successfully rayjob_test.go:160: [2026-07-13T16:03:07Z] Waiting for RayJob test-ns-x5qvg/fail-k8s-job to complete rayjob_test.go:183: [2026-07-13T16:05:34Z] Deleted RayJob test-ns-x5qvg/fail-k8s-job successfully --- PASS: TestRayJob/Failing_submitter_K8s_Job (146.80s) === RUN TestRayJob/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. rayjob_test.go:197: [2026-07-13T16:05:34Z] Created RayJob test-ns-x5qvg/stop successfully rayjob_test.go:199: [2026-07-13T16:05:34Z] Waiting for RayJob test-ns-x5qvg/stop to be 'Running' rayjob_test.go:203: [2026-07-13T16:07:30Z] Waiting for RayJob test-ns-x5qvg/stop to be 'Complete' rayjob_test.go:213: [2026-07-13T16:08:07Z] Deleted RayJob test-ns-x5qvg/stop successfully --- PASS: TestRayJob/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. (153.85s) === RUN TestRayJob/RuntimeEnvYAML_is_not_a_valid_YAML_string rayjob_test.go:225: [2026-07-13T16:08:07Z] Created RayJob test-ns-x5qvg/invalid-yamlstr successfully --- PASS: TestRayJob/RuntimeEnvYAML_is_not_a_valid_YAML_string (5.01s) === RUN TestRayJob/RayJob_has_passed_ActiveDeadlineSeconds rayjob_test.go:244: [2026-07-13T16:08:12Z] Created RayJob test-ns-x5qvg/long-running successfully rayjob_test.go:247: [2026-07-13T16:08:12Z] Waiting for RayJob test-ns-x5qvg/long-running to be 'Complete' --- PASS: TestRayJob/RayJob_has_passed_ActiveDeadlineSeconds (5.05s) === RUN TestRayJob/RayJob_should_be_created,_but_not_updated_when_managed_externally rayjob_test.go:270: [2026-07-13T16:08:17Z] Created RayJob test-ns-x5qvg/managed-externally successfully rayjob_test.go:296: [2026-07-13T16:08:17Z] Deleted RayJob test-ns-x5qvg/managed-externally successfully --- PASS: TestRayJob/RayJob_should_be_created,_but_not_updated_when_managed_externally (0.04s) test.go:114: [2026-07-13T16:08:17Z] Retrieving Pod Container test-ns-x5qvg/long-running-rfrb2-head-q8chr/ray-head logs test.go:114: [2026-07-13T16:08:18Z] Error getting logs from container test-ns-x5qvg/long-running-rfrb2-head-q8chr/ray-head test.go:114: [2026-07-13T16:08:18Z] Retrieving Pod Container test-ns-x5qvg/long-running-rfrb2-head-q8chr/kube-rbac-proxy logs test.go:114: [2026-07-13T16:08:18Z] Error getting logs from container test-ns-x5qvg/long-running-rfrb2-head-q8chr/kube-rbac-proxy test.go:114: [2026-07-13T16:08:18Z] Retrieving Pod Container test-ns-x5qvg/long-running-rfrb2-small-group-worker-7gn25/ray-worker logs test.go:114: [2026-07-13T16:08:18Z] Error getting logs from container test-ns-x5qvg/long-running-rfrb2-small-group-worker-7gn25/ray-worker test.go:102: [2026-07-13T16:08:18Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-13T16:08:18Z] Output directory has been created at: /tmp/TestRayJob502570406/001 --- PASS: TestRayJob (522.99s) === CONT TestRayJobWithClusterSelector rayjob_cluster_selector_test.go:109: [2026-07-13T16:08:18Z] Created RayJob test-ns-qbjjb/managed-externally successfully --- PASS: TestRayJobWithClusterSelector/RayJob_should_be_created_but_not_to_be_updated_when_managed_externally (1124.36s) === RUN TestRayJobWithClusterSelector/RayJob_should_not_be_created_due_to_managedBy_invalid_value --- PASS: TestRayJobWithClusterSelector/RayJob_should_not_be_created_due_to_managedBy_invalid_value (0.01s) === CONT TestRayJobWithClusterSelector/Successful_RayJob rayjob_cluster_selector_test.go:57: [2026-07-13T16:08:21Z] Created RayJob test-ns-qbjjb/counter successfully rayjob_cluster_selector_test.go:59: [2026-07-13T16:08:21Z] Waiting for RayJob test-ns-qbjjb/counter to complete --- PASS: TestRayJobWithClusterSelector/Successful_RayJob (18.10s) === CONT TestRayJobWithClusterSelector/Failing_RayJob rayjob_cluster_selector_test.go:81: [2026-07-13T16:08:39Z] Created RayJob test-ns-qbjjb/fail successfully rayjob_cluster_selector_test.go:83: [2026-07-13T16:08:39Z] Waiting for RayJob test-ns-qbjjb/fail to complete --- PASS: TestRayJobWithClusterSelector/Failing_RayJob (18.11s) test.go:114: [2026-07-13T16:08:57Z] Retrieving Pod Container test-ns-qbjjb/counter-xlnb5/ray-job-submitter logs test.go:102: [2026-07-13T16:08:57Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-13T16:08:57Z] Output directory has been created at: /tmp/TestRayJobWithClusterSelector3701676282/001 test.go:114: [2026-07-13T16:08:57Z] Retrieving Pod Container test-ns-qbjjb/fail-bmtsd/ray-job-submitter logs test.go:114: [2026-07-13T16:08:57Z] Retrieving Pod Container test-ns-qbjjb/raycluster-head-fdvcn/ray-head logs test.go:114: [2026-07-13T16:08:57Z] Retrieving Pod Container test-ns-qbjjb/raycluster-head-fdvcn/kube-rbac-proxy logs test.go:114: [2026-07-13T16:08:57Z] Retrieving Pod Container test-ns-qbjjb/raycluster-small-group-worker-96vng/ray-worker logs --- PASS: TestRayJobWithClusterSelector (35.29s) PASS ok github.com/ray-project/kuberay/ray-operator/test/e2e 1533.181s === RUN TestRayClusterAutoscaler === RUN TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaling_enabled raycluster_autoscaler_test.go:30: [2026-07-13T16:09:00Z] Created ConfigMap test-ns-9wtq2/scripts successfully raycluster_autoscaler_test.go:50: [2026-07-13T16:09:00Z] Created RayCluster test-ns-9wtq2/ray-cluster successfully raycluster_autoscaler_test.go:59: [2026-07-13T16:09:26Z] Found head pod test-ns-9wtq2/ray-cluster-head-xmqjl core.go:87: [2026-07-13T16:09:26Z] Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor1] core.go:100: [2026-07-13T16:09:27Z] Command stdout: core.go:101: [2026-07-13T16:09:27Z] Command stderr: 2026-07-13 16:09:26,636 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-13 16:09:26,639 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.133.0.40:6379... 2026-07-13 16:09:26,657 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at ray-cluster-head-svc.test-ns-9wtq2.svc.cluster.local:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( core.go:87: [2026-07-13T16:09:31Z] Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor2] core.go:100: [2026-07-13T16:09:33Z] Command stdout: core.go:101: [2026-07-13T16:09:33Z] Command stderr: 2026-07-13 16:09:32,413 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-13 16:09:32,415 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.133.0.40:6379... 2026-07-13 16:09:32,433 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at ray-cluster-head-svc.test-ns-9wtq2.svc.cluster.local:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( core.go:87: [2026-07-13T16:09:36Z] Executing command: [python /home/ray/test_scripts/terminate_detached_actor.py actor1] core.go:100: [2026-07-13T16:09:38Z] Command stdout: core.go:101: [2026-07-13T16:09:38Z] Command stderr: 2026-07-13 16:09:37,227 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-13 16:09:37,231 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.133.0.40:6379... 2026-07-13 16:09:37,250 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at ray-cluster-head-svc.test-ns-9wtq2.svc.cluster.local:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( core.go:87: [2026-07-13T16:12:19Z] Executing command: [python /home/ray/test_scripts/terminate_detached_actor.py actor2] core.go:100: [2026-07-13T16:12:20Z] Command stdout: core.go:101: [2026-07-13T16:12:20Z] Command stderr: 2026-07-13 16:12:19,536 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-13 16:12:19,540 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.133.0.40:6379... 2026-07-13 16:12:19,559 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at ray-cluster-head-svc.test-ns-9wtq2.svc.cluster.local:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( test.go:114: [2026-07-13T16:13:29Z] Retrieving Pod Container test-ns-9wtq2/ray-cluster-head-xmqjl/ray-head logs test.go:102: [2026-07-13T16:13:29Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-13T16:13:29Z] Output directory has been created at: /tmp/TestRayClusterAutoscalerCreate_a_RayCluster_with_autoscaling_en1674117069/001 test.go:114: [2026-07-13T16:13:29Z] Retrieving Pod Container test-ns-9wtq2/ray-cluster-head-xmqjl/autoscaler logs test.go:114: [2026-07-13T16:13:29Z] Retrieving Pod Container test-ns-9wtq2/ray-cluster-head-xmqjl/kube-rbac-proxy logs test.go:114: [2026-07-13T16:13:29Z] Retrieving Pod Container test-ns-9wtq2/ray-cluster-small-group-worker-tqz5h/ray-worker logs --- PASS: TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaling_enabled (270.03s) === RUN TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaler_v2_enabled raycluster_autoscaler_test.go:30: [2026-07-13T16:13:29Z] Created ConfigMap test-ns-4j8fw/scripts successfully raycluster_autoscaler_test.go:50: [2026-07-13T16:13:30Z] Created RayCluster test-ns-4j8fw/ray-cluster successfully raycluster_autoscaler_test.go:59: [2026-07-13T16:14:01Z] Found head pod test-ns-4j8fw/ray-cluster-head-f82qs core.go:87: [2026-07-13T16:14:01Z] Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor1] core.go:100: [2026-07-13T16:14:02Z] Command stdout: core.go:101: [2026-07-13T16:14:02Z] Command stderr: 2026-07-13 16:14:01,660 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-13 16:14:01,662 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.134.0.39:6379... 2026-07-13 16:14:01,681 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at ray-cluster-head-svc.test-ns-4j8fw.svc.cluster.local:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( core.go:87: [2026-07-13T16:14:08Z] Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor2] core.go:100: [2026-07-13T16:14:09Z] Command stdout: core.go:101: [2026-07-13T16:14:09Z] Command stderr: 2026-07-13 16:14:08,505 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-13 16:14:08,507 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.134.0.39:6379... 2026-07-13 16:14:08,526 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at ray-cluster-head-svc.test-ns-4j8fw.svc.cluster.local:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( core.go:87: [2026-07-13T16:14:12Z] Executing command: [python /home/ray/test_scripts/terminate_detached_actor.py actor1] core.go:100: [2026-07-13T16:14:14Z] Command stdout: core.go:101: [2026-07-13T16:14:14Z] Command stderr: 2026-07-13 16:14:13,371 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-13 16:14:13,375 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.134.0.39:6379... 2026-07-13 16:14:13,395 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at ray-cluster-head-svc.test-ns-4j8fw.svc.cluster.local:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( core.go:87: [2026-07-13T16:16:45Z] Executing command: [python /home/ray/test_scripts/terminate_detached_actor.py actor2] core.go:100: [2026-07-13T16:16:46Z] Command stdout: core.go:101: [2026-07-13T16:16:46Z] Command stderr: 2026-07-13 16:16:45,663 INFO worker.py:1672 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-07-13 16:16:45,668 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: 10.134.0.39:6379... 2026-07-13 16:16:45,687 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at ray-cluster-head-svc.test-ns-4j8fw.svc.cluster.local:8265  /opt/app-root/lib64/python3.12/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( test.go:114: [2026-07-13T16:17:56Z] Retrieving Pod Container test-ns-4j8fw/ray-cluster-head-f82qs/ray-head logs test.go:102: [2026-07-13T16:17:56Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-07-13T16:17:56Z] Output directory has been created at: /tmp/TestRayClusterAutoscalerCreate_a_RayCluster_with_autoscaler_v2_106829772/001 test.go:114: [2026-07-13T16:17:56Z] Retrieving Pod Container test-ns-4j8fw/ray-cluster-head-f82qs/autoscaler logs test.go:114: [2026-07-13T16:17:56Z] Retrieving Pod Container test-ns-4j8fw/ray-cluster-head-f82qs/kube-rbac-proxy logs test.go:114: [2026-07-13T16:17:56Z] Retrieving Pod Container test-ns-4j8fw/ray-cluster-small-group-worker-2hvgl/ray-worker logs --- PASS: TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaler_v2_enabled (266.16s) --- PASS: TestRayClusterAutoscaler (536.18s) PASS ok github.com/ray-project/kuberay/ray-operator/test/e2eautoscaler 536.204s DONE 26 tests in 2072.124s