Adding Tier1 kuberay e2e tests (including autoscaler, packages: ./test/e2erayjob ./test/e2eautoscaler) Adding Smoke kuberay e2e tests (packages: ./test/e2e) Selected tiers: Tier1 Smoke Test packages: ./test/e2erayjob ./test/e2eautoscaler ./test/e2e Running e2e tests matching: ^(TestRayJobWithClusterSelector|TestRayJob|TestRayJobSuspend|TestRayJobLightWeightMode|TestRayClusterAutoscaler|TestRayClusterAuthenticationRhoai|TestRayClusterAuthOptions|TestRayClusterRHOAIImages)$ Test timeout: 60m go: downloading go1.26.0 (linux/amd64) go: downloading github.com/onsi/gomega v1.39.1 go: downloading github.com/stretchr/testify v1.11.1 go: downloading k8s.io/api v0.36.0 go: downloading k8s.io/apimachinery v0.36.0 go: downloading k8s.io/client-go v0.36.0 go: downloading github.com/openshift/api v0.0.0-20251214014457-bfa868a22401 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2 go: downloading sigs.k8s.io/controller-runtime v0.24.1 go: downloading sigs.k8s.io/gateway-api v1.4.1 go: downloading github.com/onsi/ginkgo/v2 v2.28.1 go: downloading github.com/prometheus/client_golang v1.23.2 go: downloading github.com/go-logr/logr v1.4.3 go: downloading k8s.io/apiserver v0.36.0 go: downloading k8s.io/component-base v0.36.0 go: downloading github.com/Masterminds/semver/v3 v3.4.0 go: downloading sigs.k8s.io/yaml v1.6.0 go: downloading github.com/google/go-cmp v0.7.0 go: downloading sigs.k8s.io/randfill v1.0.0 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.140.0 go: downloading github.com/cert-manager/cert-manager v1.19.4 go: downloading github.com/orcaman/concurrent-map/v2 v2.0.1 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading k8s.io/kube-openapi v0.0.0-20260317180543-43fb72c5454a go: downloading sigs.k8s.io/structured-merge-diff/v6 v6.3.2 go: downloading sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 go: downloading github.com/google/gnostic-models v0.7.1 go: downloading google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading golang.org/x/net v0.57.0 go: downloading github.com/evanphx/json-patch/v5 v5.9.11 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/client_model v0.6.2 go: downloading github.com/prometheus/common v0.67.5 go: downloading github.com/prometheus/procfs v0.19.2 go: downloading golang.org/x/term v0.45.0 go: downloading github.com/spf13/pflag v1.0.10 go: downloading github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 go: downloading k8s.io/streaming v0.36.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading go.yaml.in/yaml/v2 v2.4.3 go: downloading go.yaml.in/yaml/v3 v3.0.4 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/fxamacker/cbor/v2 v2.9.0 go: downloading golang.org/x/oauth2 v0.34.0 go: downloading golang.org/x/time v0.14.0 go: downloading gomodules.xyz/jsonpatch/v2 v2.5.0 go: downloading k8s.io/apiextensions-apiserver v0.36.0 go: downloading golang.org/x/sys v0.47.0 go: downloading github.com/moby/spdystream v0.5.1 go: downloading golang.org/x/text v0.40.0 go: downloading volcano.sh/apis v1.13.0 go: downloading sigs.k8s.io/scheduler-plugins v0.33.5 go: downloading github.com/google/uuid v1.6.0 go: downloading golang.org/x/sync v0.22.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee go: downloading github.com/go-openapi/jsonreference v0.21.4 go: downloading github.com/go-openapi/swag v0.25.4 go: downloading github.com/x448/float16 v0.8.4 go: downloading github.com/fsnotify/fsnotify v1.9.0 go: downloading golang.org/x/tools v0.47.0 go: downloading github.com/blang/semver/v4 v4.0.0 go: downloading go.opentelemetry.io/otel/trace v1.41.0 go: downloading github.com/emicklei/go-restful/v3 v3.13.0 go: downloading go.opentelemetry.io/otel v1.41.0 go: downloading github.com/go-openapi/jsonpointer v0.22.4 go: downloading github.com/go-openapi/swag/cmdutils v0.25.4 go: downloading github.com/go-openapi/swag/conv v0.25.4 go: downloading github.com/go-openapi/swag/fileutils v0.25.4 go: downloading github.com/go-openapi/swag/jsonname v0.25.4 go: downloading github.com/go-openapi/swag/jsonutils v0.25.4 go: downloading github.com/go-openapi/swag/loading v0.25.4 go: downloading github.com/go-openapi/swag/mangling v0.25.4 go: downloading github.com/go-openapi/swag/netutils v0.25.4 go: downloading github.com/go-openapi/swag/stringutils v0.25.4 go: downloading github.com/go-openapi/swag/typeutils v0.25.4 go: downloading github.com/go-openapi/swag/yamlutils v0.25.4 go: downloading gopkg.in/evanphx/json-patch.v4 v4.13.0 go: downloading golang.org/x/mod v0.37.0 === RUN TestRayJobWithClusterSelector rayjob_cluster_selector_test.go:27: [2026-11-03T14:58:40Z] Created ConfigMap test-ns-g9m7p/jobs successfully rayjob_cluster_selector_test.go:35: [2026-11-03T14:58:41Z] Created RayCluster test-ns-g9m7p/raycluster successfully rayjob_cluster_selector_test.go:37: [2026-11-03T14:58:41Z] Waiting for RayCluster test-ns-g9m7p/raycluster to become ready === RUN TestRayJobWithClusterSelector/Successful_RayJob === PAUSE TestRayJobWithClusterSelector/Successful_RayJob === RUN TestRayJobWithClusterSelector/Failing_RayJob === PAUSE TestRayJobWithClusterSelector/Failing_RayJob === RUN TestRayJobWithClusterSelector/RayJob_should_be_created_but_not_to_be_updated_when_managed_externally rayjob_cluster_selector_test.go:109: [2026-11-03T15:02:35Z] Created RayJob test-ns-g9m7p/managed-externally successfully --- PASS: TestRayJobWithClusterSelector/RayJob_should_be_created_but_not_to_be_updated_when_managed_externally (3.02s) === RUN TestRayJobWithClusterSelector/RayJob_should_not_be_created_due_to_managedBy_invalid_value --- PASS: TestRayJobWithClusterSelector/RayJob_should_not_be_created_due_to_managedBy_invalid_value (0.02s) === CONT TestRayJobWithClusterSelector/Successful_RayJob rayjob_cluster_selector_test.go:57: [2026-11-03T15:02:38Z] Created RayJob test-ns-g9m7p/counter successfully rayjob_cluster_selector_test.go:59: [2026-11-03T15:02:38Z] Waiting for RayJob test-ns-g9m7p/counter to complete --- PASS: TestRayJobWithClusterSelector/Successful_RayJob (15.31s) === CONT TestRayJobWithClusterSelector/Failing_RayJob rayjob_cluster_selector_test.go:81: [2026-11-03T15:02:53Z] Created RayJob test-ns-g9m7p/fail successfully rayjob_cluster_selector_test.go:83: [2026-11-03T15:02:53Z] Waiting for RayJob test-ns-g9m7p/fail to complete --- PASS: TestRayJobWithClusterSelector/Failing_RayJob (12.26s) test.go:114: [2026-11-03T15:03:06Z] Retrieving Pod Container test-ns-g9m7p/counter-h2lp7/ray-job-submitter logs test.go:102: [2026-11-03T15:03:06Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-11-03T15:03:06Z] Output directory has been created at: /tmp/kuberay-test-home/tmp/TestRayJobWithClusterSelector1050742563/001 test.go:114: [2026-11-03T15:03:06Z] Retrieving Pod Container test-ns-g9m7p/fail-qqc5v/ray-job-submitter logs test.go:114: [2026-11-03T15:03:06Z] Retrieving Pod Container test-ns-g9m7p/raycluster-head-b4s9s/ray-head logs test.go:114: [2026-11-03T15:03:06Z] Retrieving Pod Container test-ns-g9m7p/raycluster-head-b4s9s/kube-rbac-proxy logs test.go:114: [2026-11-03T15:03:06Z] Retrieving Pod Container test-ns-g9m7p/raycluster-small-group-worker-qrhm7/ray-worker logs --- PASS: TestRayJobWithClusterSelector (238.29s) === RUN TestRayJobLightWeightMode rayjob_lightweight_test.go:28: [2026-11-03T15:03:06Z] Created ConfigMap test-ns-mvz6h/jobs successfully === RUN TestRayJobLightWeightMode/Successful_RayJob rayjob_lightweight_test.go:57: [2026-11-03T15:03:06Z] Created RayJob test-ns-mvz6h/counter successfully rayjob_lightweight_test.go:59: [2026-11-03T15:03:06Z] Waiting for RayJob test-ns-mvz6h/counter to complete --- PASS: TestRayJobLightWeightMode/Successful_RayJob (112.06s) === RUN TestRayJobLightWeightMode/Failing_RayJob_without_cluster_shutdown_after_finished rayjob_lightweight_test.go:91: [2026-11-03T15:04:58Z] Created RayJob test-ns-mvz6h/fail successfully rayjob_lightweight_test.go:93: [2026-11-03T15:04:58Z] Waiting for RayJob test-ns-mvz6h/fail to complete rayjob_lightweight_test.go:117: [2026-11-03T15:06:57Z] Deleted RayJob test-ns-mvz6h/fail successfully --- PASS: TestRayJobLightWeightMode/Failing_RayJob_without_cluster_shutdown_after_finished (119.25s) === RUN TestRayJobLightWeightMode/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. rayjob_lightweight_test.go:137: [2026-11-03T15:06:58Z] Created RayJob test-ns-mvz6h/stop successfully rayjob_lightweight_test.go:139: [2026-11-03T15:06:58Z] Waiting for RayJob test-ns-mvz6h/stop to be 'Running' rayjob_lightweight_test.go:143: [2026-11-03T15:08:44Z] Waiting for RayJob test-ns-mvz6h/stop to be 'Complete' rayjob_lightweight_test.go:153: [2026-11-03T15:09:09Z] Deleted RayJob test-ns-mvz6h/stop successfully --- PASS: TestRayJobLightWeightMode/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. (131.42s) === RUN TestRayJobLightWeightMode/Successful_RayJob_in_HTTP_mode_with_auth_token rayjob_lightweight_test.go:172: [2026-11-03T15:09:09Z] Created RayJob test-ns-mvz6h/counter-auth successfully with auth token rayjob_lightweight_test.go:175: [2026-11-03T15:09:09Z] Waiting for RayCluster to be created rayjob_lightweight_test.go:185: [2026-11-03T15:09:10Z] Waiting for RayCluster test-ns-mvz6h/counter-auth-kk8bt to become ready rayjob_lightweight_test.go:208: [2026-11-03T15:11:04Z] Waiting for RayJob test-ns-mvz6h/counter-auth to complete rayjob_lightweight_test.go:220: [2026-11-03T15:11:11Z] RayJob test-ns-mvz6h/counter-auth completed successfully with auth token --- PASS: TestRayJobLightWeightMode/Successful_RayJob_in_HTTP_mode_with_auth_token (122.34s) test.go:114: [2026-11-03T15:11:11Z] Retrieving Pod Container test-ns-mvz6h/counter-auth-kk8bt-head-27gn8/ray-head logs test.go:102: [2026-11-03T15:11:12Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-11-03T15:11:12Z] Output directory has been created at: /tmp/kuberay-test-home/tmp/TestRayJobLightWeightMode1962169790/001 test.go:114: [2026-11-03T15:11:12Z] Retrieving Pod Container test-ns-mvz6h/counter-auth-kk8bt-head-27gn8/kube-rbac-proxy logs test.go:114: [2026-11-03T15:11:12Z] Retrieving Pod Container test-ns-mvz6h/counter-auth-kk8bt-small-group-worker-k56zq/ray-worker logs --- PASS: TestRayJobLightWeightMode (485.59s) === RUN TestRayJobSuspend rayjob_suspend_test.go:27: [2026-11-03T15:11:12Z] Created ConfigMap test-ns-5lqjm/jobs successfully === RUN TestRayJobSuspend/Suspend_the_RayJob_when_its_status_is_'Running',_and_then_resume_it. rayjob_suspend_test.go:41: [2026-11-03T15:11:12Z] Created RayJob test-ns-5lqjm/long-running successfully rayjob_suspend_test.go:43: [2026-11-03T15:11:12Z] Waiting for RayJob test-ns-5lqjm/long-running to be 'Running' rayjob_suspend_test.go:47: [2026-11-03T15:13:13Z] Suspend the RayJob test-ns-5lqjm/long-running rayjob_suspend_test.go:52: [2026-11-03T15:13:13Z] Waiting for RayJob test-ns-5lqjm/long-running to be 'Suspended' rayjob_suspend_test.go:68: [2026-11-03T15:13:14Z] Resume the RayJob by updating `suspend` to false. rayjob_suspend_test.go:78: [2026-11-03T15:14:52Z] Deleted RayJob test-ns-5lqjm/long-running successfully --- PASS: TestRayJobSuspend/Suspend_the_RayJob_when_its_status_is_'Running',_and_then_resume_it. (220.08s) === RUN TestRayJobSuspend/Create_a_suspended_RayJob,_and_then_resume_it. rayjob_suspend_test.go:97: [2026-11-03T15:14:52Z] Created RayJob test-ns-5lqjm/counter successfully rayjob_suspend_test.go:99: [2026-11-03T15:14:52Z] Waiting for RayJob test-ns-5lqjm/counter to be 'Suspended' rayjob_suspend_test.go:103: [2026-11-03T15:14:53Z] Resume the RayJob by updating `suspend` to false. rayjob_suspend_test.go:108: [2026-11-03T15:14:53Z] Waiting for RayJob test-ns-5lqjm/counter to complete rayjob_suspend_test.go:123: [2026-11-03T15:17:12Z] Deleted RayJob test-ns-5lqjm/counter successfully --- PASS: TestRayJobSuspend/Create_a_suspended_RayJob,_and_then_resume_it. (142.72s) test.go:102: [2026-11-03T15:17:15Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-11-03T15:17:15Z] Output directory has been created at: /tmp/kuberay-test-home/tmp/TestRayJobSuspend544196660/001 --- PASS: TestRayJobSuspend (362.91s) === RUN TestRayJob rayjob_test.go:33: [2026-11-03T15:17:15Z] Created ConfigMap test-ns-kbx4w/jobs successfully === RUN TestRayJob/Successful_RayJob rayjob_test.go:50: [2026-11-03T15:17:15Z] Created RayJob test-ns-kbx4w/counter successfully rayjob_test.go:60: [2026-11-03T15:19:14Z] Waiting for RayJob test-ns-kbx4w/counter to complete rayjob_test.go:76: [2026-11-03T15:19:33Z] Checking that the RayJob status info has been set correctly. rayjob_test.go:93: [2026-11-03T15:19:34Z] Update `suspend` to true. However, since the RayJob is completed, the status should not be updated to `Suspended`. rayjob_test.go:103: [2026-11-03T15:20:04Z] Deleted RayJob test-ns-kbx4w/counter successfully --- PASS: TestRayJob/Successful_RayJob (169.46s) === RUN TestRayJob/Failing_RayJob_without_cluster_shutdown_after_finished rayjob_test.go:117: [2026-11-03T15:20:04Z] Created RayJob test-ns-kbx4w/fail successfully rayjob_test.go:119: [2026-11-03T15:20:04Z] Waiting for RayJob test-ns-kbx4w/fail to complete rayjob_test.go:142: [2026-11-03T15:22:19Z] Deleted RayJob test-ns-kbx4w/fail successfully --- PASS: TestRayJob/Failing_RayJob_without_cluster_shutdown_after_finished (135.61s) === RUN TestRayJob/Failing_submitter_K8s_Job rayjob_test.go:170: [2026-11-03T15:22:20Z] Created RayJob test-ns-kbx4w/fail-k8s-job successfully rayjob_test.go:172: [2026-11-03T15:22:20Z] Waiting for RayJob test-ns-kbx4w/fail-k8s-job to complete rayjob_test.go:195: [2026-11-03T15:24:53Z] Deleted RayJob test-ns-kbx4w/fail-k8s-job successfully --- PASS: TestRayJob/Failing_submitter_K8s_Job (152.97s) === RUN TestRayJob/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. rayjob_test.go:209: [2026-11-03T15:24:53Z] Created RayJob test-ns-kbx4w/stop successfully rayjob_test.go:211: [2026-11-03T15:24:53Z] Waiting for RayJob test-ns-kbx4w/stop to be 'Running' rayjob_test.go:215: [2026-11-03T15:26:52Z] Waiting for RayJob test-ns-kbx4w/stop to be 'Complete' rayjob_test.go:225: [2026-11-03T15:27:29Z] Deleted RayJob test-ns-kbx4w/stop successfully --- PASS: TestRayJob/Should_transition_to_'Complete'_if_the_Ray_job_has_stopped. (155.91s) === RUN TestRayJob/RuntimeEnvYAML_is_not_a_valid_YAML_string rayjob_test.go:237: [2026-11-03T15:27:29Z] Created RayJob test-ns-kbx4w/invalid-yamlstr successfully --- PASS: TestRayJob/RuntimeEnvYAML_is_not_a_valid_YAML_string (1.06s) === RUN TestRayJob/RayJob_name_too_long_with_48_characters rayjob_test.go:252: [2026-11-03T15:27:30Z] Created RayJob test-ns-kbx4w/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa successfully --- PASS: TestRayJob/RayJob_name_too_long_with_48_characters (1.06s) === RUN TestRayJob/RayJob_has_passed_ActiveDeadlineSeconds rayjob_test.go:271: [2026-11-03T15:27:31Z] Created RayJob test-ns-kbx4w/long-running successfully rayjob_test.go:274: [2026-11-03T15:27:31Z] Waiting for RayJob test-ns-kbx4w/long-running to be 'Complete' --- PASS: TestRayJob/RayJob_has_passed_ActiveDeadlineSeconds (8.21s) === RUN TestRayJob/RayJob_controller_recreates_the_head_Pod_if_it_is_deleted_while_the_job_is_running rayjob_test.go:293: [2026-11-03T15:27:39Z] Created RayJob test-ns-kbx4w/delete-head-after-submit successfully rayjob_test.go:296: [2026-11-03T15:27:39Z] Waiting for RayJob test-ns-kbx4w/delete-head-after-submit to be 'Running' rayjob_test.go:309: [2026-11-03T15:29:34Z] Deleting head Pod test-ns-kbx4w/delete-head-after-submit-gg2wh-head-b99bx for RayCluster delete-head-after-submit-gg2wh rayjob_test.go:338: [2026-11-03T15:31:53Z] Deleted RayJob test-ns-kbx4w/delete-head-after-submit successfully --- PASS: TestRayJob/RayJob_controller_recreates_the_head_Pod_if_it_is_deleted_while_the_job_is_running (253.85s) === RUN TestRayJob/RayJob_should_be_created,_but_not_updated_when_managed_externally rayjob_test.go:357: [2026-11-03T15:31:53Z] Created RayJob test-ns-kbx4w/managed-externally successfully rayjob_test.go:383: [2026-11-03T15:31:53Z] Deleted RayJob test-ns-kbx4w/managed-externally successfully --- PASS: TestRayJob/RayJob_should_be_created,_but_not_updated_when_managed_externally (0.13s) === RUN TestRayJob/RayJob_has_exceed_SubmitterFinishedTimeout rayjob_test.go:396: [2026-11-03T15:31:53Z] Created RayJob test-ns-kbx4w/submitter-timeout successfully rayjob_test.go:399: [2026-11-03T15:31:53Z] Waiting for RayJob test-ns-kbx4w/submitter-timeout deployment status to be 'Running' rayjob_test.go:402: [2026-11-03T15:33:42Z] Waiting for Ray job test-ns-kbx4w/submitter-timeout to be actually running in Ray cluster rayjob_test.go:407: [2026-11-03T15:33:53Z] Waiting for submitter job to be created rayjob_test.go:412: [2026-11-03T15:33:53Z] Updating submitter job status to complete rayjob_test.go:441: [2026-11-03T15:33:53Z] Successfully marked submitter job as completed at 2026-11-03 15:33:53.90555866 +0000 UTC m=+2113.093072696 rayjob_test.go:447: [2026-11-03T15:33:53Z] Waiting for RayJob test-ns-kbx4w/submitter-timeout to exceed SubmitterFinishedTimeout --- PASS: TestRayJob/RayJob_has_exceed_SubmitterFinishedTimeout (151.03s) === RUN TestRayJob/RayCluster_status_update_propagates_to_RayJob rayjob_test.go:477: [2026-11-03T15:34:24Z] Created RayJob test-ns-kbx4w/cluster-status-update successfully rayjob_test.go:505: [2026-11-03T15:36:29Z] Deleted RayJob test-ns-kbx4w/cluster-status-update successfully --- PASS: TestRayJob/RayCluster_status_update_propagates_to_RayJob (125.45s) === RUN TestRayJob/Successful_RayJob_in_K8s_Job_mode_with_auth_token rayjob_test.go:525: [2026-11-03T15:36:30Z] Created RayJob test-ns-kbx4w/counter-auth successfully with auth token rayjob_test.go:528: [2026-11-03T15:36:30Z] Waiting for RayCluster to be created rayjob_test.go:538: [2026-11-03T15:36:31Z] Waiting for RayCluster test-ns-kbx4w/counter-auth-qmw8r to become ready rayjob_test.go:562: [2026-11-03T15:38:30Z] Waiting for submitter Job to be created rayjob_test.go:568: [2026-11-03T15:38:30Z] Waiting for submitter Job pod to be created rayjob_test.go:578: [2026-11-03T15:38:30Z] Waiting for RayJob test-ns-kbx4w/counter-auth to complete rayjob_test.go:590: [2026-11-03T15:38:50Z] RayJob test-ns-kbx4w/counter-auth completed successfully with auth token --- PASS: TestRayJob/Successful_RayJob_in_K8s_Job_mode_with_auth_token (140.83s) === RUN TestRayJob/RayJob_PreRunningDeadlineSeconds_expires_during_Initializing_state rayjob_test.go:609: [2026-11-03T15:38:50Z] Created RayJob test-ns-kbx4w/ttl-initializing successfully rayjob_test.go:616: [2026-11-03T15:38:51Z] Waiting for RayJob test-ns-kbx4w/ttl-initializing to be 'Failed' --- PASS: TestRayJob/RayJob_PreRunningDeadlineSeconds_expires_during_Initializing_state (7.21s) === RUN TestRayJob/RayJob_PreRunningDeadlineSeconds_expires_during_Waiting_state rayjob_test.go:653: [2026-11-03T15:40:52Z] Waiting for RayJob test-ns-kbx4w/ttl-waiting to be 'Failed' --- PASS: TestRayJob/RayJob_PreRunningDeadlineSeconds_expires_during_Waiting_state (115.21s) test.go:114: [2026-11-03T15:40:53Z] Retrieving Pod Container test-ns-kbx4w/counter-auth-k9kkt/ray-job-submitter logs test.go:102: [2026-11-03T15:40:53Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-11-03T15:40:53Z] Output directory has been created at: /tmp/kuberay-test-home/tmp/TestRayJob4161490720/001 test.go:114: [2026-11-03T15:40:53Z] Retrieving Pod Container test-ns-kbx4w/submitter-timeout-khdxd/ray-job-submitter logs test.go:114: [2026-11-03T15:40:53Z] Retrieving Pod Container test-ns-kbx4w/ttl-waiting-kjf64-head-dvbng/ray-head logs test.go:114: [2026-11-03T15:40:53Z] Retrieving Pod Container test-ns-kbx4w/ttl-waiting-kjf64-head-dvbng/kube-rbac-proxy logs test.go:114: [2026-11-03T15:40:53Z] Retrieving Pod Container test-ns-kbx4w/ttl-waiting-kjf64-small-group-worker-blcp8/ray-worker logs --- PASS: TestRayJob (1418.78s) PASS ok github.com/ray-project/kuberay/ray-operator/test/e2erayjob 2533.166s === RUN TestRayClusterAutoscaler === RUN TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaling_enabled raycluster_autoscaler_test.go:30: [2026-11-03T15:40:55Z] Created ConfigMap test-ns-4jsqd/scripts successfully raycluster_autoscaler_test.go:50: [2026-11-03T15:40:56Z] Created RayCluster test-ns-4jsqd/ray-cluster successfully raycluster_autoscaler_test.go:59: [2026-11-03T15:42:46Z] Found head pod test-ns-4jsqd/ray-cluster-head-9lzd6 core.go:88: [2026-11-03T15:42:46Z] Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor1] core.go:101: [2026-11-03T15:42:48Z] Command stdout: core.go:102: [2026-11-03T15:42:48Z] Command stderr: 2026-11-03 15:42:47,049 INFO worker.py:1683 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-11-03 15:42:47,051 INFO worker.py:1833 -- Connecting to existing Ray cluster at address: 10.134.0.56:6379... 2026-11-03 15:42:47,076 INFO worker.py:2015 -- Connected to Ray cluster. View the dashboard at http://10.134.0.56:8265  core.go:88: [2026-11-03T15:42:54Z] Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor2] core.go:101: [2026-11-03T15:42:56Z] Command stdout: core.go:102: [2026-11-03T15:42:56Z] Command stderr: 2026-11-03 15:42:55,403 INFO worker.py:1683 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-11-03 15:42:55,406 INFO worker.py:1833 -- Connecting to existing Ray cluster at address: 10.134.0.56:6379... 2026-11-03 15:42:55,430 INFO worker.py:2015 -- Connected to Ray cluster. View the dashboard at http://10.134.0.56:8265  core.go:88: [2026-11-03T15:43:00Z] Executing command: [python /home/ray/test_scripts/terminate_detached_actor.py actor1] core.go:101: [2026-11-03T15:43:02Z] Command stdout: core.go:102: [2026-11-03T15:43:02Z] Command stderr: 2026-11-03 15:43:01,653 INFO worker.py:1683 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-11-03 15:43:01,657 INFO worker.py:1833 -- Connecting to existing Ray cluster at address: 10.134.0.56:6379... 2026-11-03 15:43:01,682 INFO worker.py:2015 -- Connected to Ray cluster. View the dashboard at http://10.134.0.56:8265  core.go:88: [2026-11-03T15:45:39Z] Executing command: [python /home/ray/test_scripts/terminate_detached_actor.py actor2] core.go:101: [2026-11-03T15:45:41Z] Command stdout: core.go:102: [2026-11-03T15:45:41Z] Command stderr: 2026-11-03 15:45:40,427 INFO worker.py:1683 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-11-03 15:45:40,431 INFO worker.py:1833 -- Connecting to existing Ray cluster at address: 10.134.0.56:6379... 2026-11-03 15:45:40,469 INFO worker.py:2015 -- Connected to Ray cluster. View the dashboard at http://10.134.0.56:8265  test.go:114: [2026-11-03T15:46:50Z] Retrieving Pod Container test-ns-4jsqd/ray-cluster-head-9lzd6/ray-head logs test.go:102: [2026-11-03T15:46:50Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-11-03T15:46:50Z] Output directory has been created at: /tmp/kuberay-test-home/tmp/TestRayClusterAutoscalerCreate_a_RayCluster_with_autoscaling_en2088688379/001 test.go:114: [2026-11-03T15:46:50Z] Retrieving Pod Container test-ns-4jsqd/ray-cluster-head-9lzd6/autoscaler logs test.go:114: [2026-11-03T15:46:50Z] Retrieving Pod Container test-ns-4jsqd/ray-cluster-head-9lzd6/kube-rbac-proxy logs test.go:114: [2026-11-03T15:46:51Z] Retrieving Pod Container test-ns-4jsqd/ray-cluster-small-group-worker-vwxmp/ray-worker logs test.go:114: [2026-11-03T15:46:51Z] Error getting logs from container test-ns-4jsqd/ray-cluster-small-group-worker-vwxmp/ray-worker --- PASS: TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaling_enabled (355.27s) === RUN TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaler_v2_enabled raycluster_autoscaler_test.go:30: [2026-11-03T15:46:51Z] Created ConfigMap test-ns-zjpgs/scripts successfully raycluster_autoscaler_test.go:50: [2026-11-03T15:46:51Z] Created RayCluster test-ns-zjpgs/ray-cluster successfully raycluster_autoscaler_test.go:59: [2026-11-03T15:48:44Z] Found head pod test-ns-zjpgs/ray-cluster-head-jhq58 core.go:88: [2026-11-03T15:48:44Z] Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor1] core.go:101: [2026-11-03T15:48:46Z] Command stdout: core.go:102: [2026-11-03T15:48:46Z] Command stderr: 2026-11-03 15:48:45,386 INFO worker.py:1683 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-11-03 15:48:45,388 INFO worker.py:1833 -- Connecting to existing Ray cluster at address: 10.132.0.54:6379... 2026-11-03 15:48:45,412 INFO worker.py:2015 -- Connected to Ray cluster. View the dashboard at http://10.132.0.54:8265  core.go:88: [2026-11-03T15:48:52Z] Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor2] core.go:101: [2026-11-03T15:48:55Z] Command stdout: core.go:102: [2026-11-03T15:48:55Z] Command stderr: 2026-11-03 15:48:53,731 INFO worker.py:1683 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-11-03 15:48:53,733 INFO worker.py:1833 -- Connecting to existing Ray cluster at address: 10.132.0.54:6379... 2026-11-03 15:48:53,758 INFO worker.py:2015 -- Connected to Ray cluster. View the dashboard at http://10.132.0.54:8265  core.go:88: [2026-11-03T15:49:00Z] Executing command: [python /home/ray/test_scripts/terminate_detached_actor.py actor1] core.go:101: [2026-11-03T15:49:02Z] Command stdout: core.go:102: [2026-11-03T15:49:02Z] Command stderr: 2026-11-03 15:49:01,066 INFO worker.py:1683 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-11-03 15:49:01,071 INFO worker.py:1833 -- Connecting to existing Ray cluster at address: 10.132.0.54:6379... 2026-11-03 15:49:01,097 INFO worker.py:2015 -- Connected to Ray cluster. View the dashboard at http://10.132.0.54:8265  core.go:88: [2026-11-03T15:51:36Z] Executing command: [python /home/ray/test_scripts/terminate_detached_actor.py actor2] core.go:101: [2026-11-03T15:51:38Z] Command stdout: core.go:102: [2026-11-03T15:51:38Z] Command stderr: 2026-11-03 15:51:37,760 INFO worker.py:1683 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2026-11-03 15:51:37,764 INFO worker.py:1833 -- Connecting to existing Ray cluster at address: 10.132.0.54:6379... 2026-11-03 15:51:37,789 INFO worker.py:2015 -- Connected to Ray cluster. View the dashboard at http://10.132.0.54:8265  test.go:114: [2026-11-03T15:52:48Z] Retrieving Pod Container test-ns-zjpgs/ray-cluster-head-jhq58/ray-head logs test.go:102: [2026-11-03T15:52:48Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-11-03T15:52:48Z] Output directory has been created at: /tmp/kuberay-test-home/tmp/TestRayClusterAutoscalerCreate_a_RayCluster_with_autoscaler_v2_2112597267/001 test.go:114: [2026-11-03T15:52:48Z] Retrieving Pod Container test-ns-zjpgs/ray-cluster-head-jhq58/autoscaler logs test.go:114: [2026-11-03T15:52:48Z] Retrieving Pod Container test-ns-zjpgs/ray-cluster-head-jhq58/kube-rbac-proxy logs test.go:114: [2026-11-03T15:52:48Z] Retrieving Pod Container test-ns-zjpgs/ray-cluster-small-group-worker-lc796/ray-worker logs test.go:114: [2026-11-03T15:52:48Z] Error getting logs from container test-ns-zjpgs/ray-cluster-small-group-worker-lc796/ray-worker --- PASS: TestRayClusterAutoscaler/Create_a_RayCluster_with_autoscaler_v2_enabled (357.60s) --- PASS: TestRayClusterAutoscaler (712.87s) PASS ok github.com/ray-project/kuberay/ray-operator/test/e2eautoscaler 712.888s === RUN TestRayClusterAuthOptions === RUN TestRayClusterAuthOptions/RayCluster_with_token_authentication_enabled === PAUSE TestRayClusterAuthOptions/RayCluster_with_token_authentication_enabled === CONT TestRayClusterAuthOptions/RayCluster_with_token_authentication_enabled raycluster_auth_test.go:38: [2026-11-03T15:52:51Z] Created RayCluster test-ns-rhs44/raycluster-auth-token successfully with AuthModeToken raycluster_auth_test.go:40: [2026-11-03T15:52:51Z] Waiting for RayCluster test-ns-rhs44/raycluster-auth-token to become ready === RUN TestRayClusterAuthOptions/Submit_job_with_auth_token_should_succeed raycluster_auth_test.go:65: [2026-11-03T15:54:29Z] Testing job submission WITH auth token core.go:88: [2026-11-03T15:54:29Z] Executing command: [bash -c RAY_AUTH_TOKEN=OC4Xt5aglqCmjWPY7FKTt0jZuhkif4YLYb9/lfA5maE= ray job submit --address http://127.0.0.1:8265 --submission-id test-job-with-auth-1793721269 --no-wait -- python -c 'import ray; ray.init(); print("Job with auth succeeded")'] core.go:101: [2026-11-03T15:54:32Z] Command stdout: 2026-11-03 15:54:31,150 INFO cli.py:42 -- Job submission server address: http://10.134.0.59:8265 2026-11-03 15:54:31,993 SUCC cli.py:66 -- ---------------------------------------------------------- 2026-11-03 15:54:31,993 SUCC cli.py:67 -- Job 'test-job-with-auth-1793721269' submitted successfully 2026-11-03 15:54:31,993 SUCC cli.py:68 -- ---------------------------------------------------------- 2026-11-03 15:54:31,993 INFO cli.py:330 -- Next steps 2026-11-03 15:54:31,993 INFO cli.py:331 -- Query the logs of the job: 2026-11-03 15:54:31,993 INFO cli.py:333 -- ray job logs test-job-with-auth-1793721269 2026-11-03 15:54:31,993 INFO cli.py:335 -- Query the status of the job: 2026-11-03 15:54:31,993 INFO cli.py:337 -- ray job status test-job-with-auth-1793721269 2026-11-03 15:54:31,993 INFO cli.py:339 -- Request the job to be stopped: 2026-11-03 15:54:31,993 INFO cli.py:341 -- ray job stop test-job-with-auth-1793721269 core.go:102: [2026-11-03T15:54:32Z] Command stderr: core.go:88: [2026-11-03T15:54:32Z] Executing command: [bash -c RAY_AUTH_TOKEN=OC4Xt5aglqCmjWPY7FKTt0jZuhkif4YLYb9/lfA5maE= ray job status --address http://127.0.0.1:8265 test-job-with-auth-1793721269] core.go:101: [2026-11-03T15:54:34Z] Command stdout: 2026-11-03 15:54:34,490 INFO cli.py:42 -- Job submission server address: http://10.134.0.59:8265 2026-11-03 15:54:34,494 INFO cli.py:92 -- Status for job 'test-job-with-auth-1793721269': RUNNING 2026-11-03 15:54:34,494 INFO cli.py:94 -- Status message: Job is currently running. core.go:102: [2026-11-03T15:54:34Z] Command stderr: core.go:88: [2026-11-03T15:54:35Z] Executing command: [bash -c RAY_AUTH_TOKEN=OC4Xt5aglqCmjWPY7FKTt0jZuhkif4YLYb9/lfA5maE= ray job status --address http://127.0.0.1:8265 test-job-with-auth-1793721269] core.go:101: [2026-11-03T15:54:38Z] Command stdout: 2026-11-03 15:54:37,852 INFO cli.py:42 -- Job submission server address: http://10.134.0.59:8265 2026-11-03 15:54:37,855 SUCC cli.py:66 -- --------------------------------------------- 2026-11-03 15:54:37,855 SUCC cli.py:67 -- Job 'test-job-with-auth-1793721269' succeeded 2026-11-03 15:54:37,855 SUCC cli.py:68 -- --------------------------------------------- core.go:102: [2026-11-03T15:54:38Z] Command stderr: raycluster_auth_test.go:92: [2026-11-03T15:54:38Z] Successfully submitted and verified job with auth token --- PASS: TestRayClusterAuthOptions/Submit_job_with_auth_token_should_succeed (9.08s) === RUN TestRayClusterAuthOptions/Submit_job_with_incorrect_auth_token_should_fail raycluster_auth_test.go:96: [2026-11-03T15:54:38Z] Testing job submission WITH incorrect auth token (should fail) core.go:88: [2026-11-03T15:54:38Z] Executing command: [bash -c RAY_AUTH_TOKEN=incorrect-token-12345 ray job submit --address http://127.0.0.1:8265 --submission-id test-job-bad-auth-1793721278 --no-wait -- python -c 'print("Should not run")'] core.go:101: [2026-11-03T15:54:40Z] Command stdout: core.go:102: [2026-11-03T15:54:40Z] Command stderr: [2026-11-03 15:54:40,142 W 855 855] gcs_client.cc:210: Failed to get cluster ID from GCS server: Unauthenticated: RPC error: InvalidAuthToken: Authentication token is missing or incorrect Traceback (most recent call last): File "/opt/app-root/bin/ray", line 8, in sys.exit(main()) ^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/scripts/scripts.py", line 3010, in main return cli() ^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1631, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1552, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 2032, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 2032, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1415, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 910, in invoke return callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/dashboard/modules/job/cli_utils.py", line 54, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/autoscaler/_private/cli_logger.py", line 812, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/dashboard/modules/job/cli.py", line 303, in submit client = _get_sdk_client( ^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/dashboard/modules/job/cli.py", line 35, in _get_sdk_client client = JobSubmissionClient( ^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/dashboard/modules/job/sdk.py", line 102, in __init__ api_server_url = get_address_for_submission_client(address) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/dashboard/utils.py", line 745, in get_address_for_submission_client address = ray_address_to_api_server_url(address) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/app-root/lib64/python3.12/site-packages/ray/dashboard/utils.py", line 688, in ray_address_to_api_server_url gcs_client = GcsClient(address=address) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "python/ray/_raylet.pyx", line 3200, in ray._raylet.GcsClient.__cinit__ File "python/ray/includes/gcs_client.pxi", line 73, in ray._raylet.InnerGcsClient.standalone File "python/ray/includes/common.pxi", line 144, in ray._raylet.check_status_timeout_as_rpc_error File "python/ray/includes/common.pxi", line 117, in ray._raylet.check_status ray.exceptions.AuthenticationError: RPC error: InvalidAuthToken: Authentication token is missing or incorrect. Ensure that the token for the cluster is available in a local file (e.g., ~/.ray/auth_token or via RAY_AUTH_TOKEN_PATH) or as the `RAY_AUTH_TOKEN` environment variable. To generate a token for local development, use `ray get-auth-token --generate` For remote clusters, ensure that the token is propagated to all nodes of the cluster when token authentication is enabled. For more information, see: https://docs.ray.io/en/latest/ray-security/token-auth.html raycluster_auth_test.go:113: [2026-11-03T15:54:40Z] Job submission correctly rejected with incorrect auth token --- PASS: TestRayClusterAuthOptions/Submit_job_with_incorrect_auth_token_should_fail (2.29s) --- PASS: TestRayClusterAuthOptions/RayCluster_with_token_authentication_enabled (109.33s) test.go:114: [2026-11-03T15:54:40Z] Retrieving Pod Container test-ns-rhs44/raycluster-auth-token-head-985vp/ray-head logs test.go:102: [2026-11-03T15:54:40Z] Creating ephemeral output directory as KUBERAY_TEST_OUTPUT_DIR env variable is unset test.go:105: [2026-11-03T15:54:40Z] Output directory has been created at: /tmp/kuberay-test-home/tmp/TestRayClusterAuthOptions2608091398/001 test.go:114: [2026-11-03T15:54:40Z] Retrieving Pod Container test-ns-rhs44/raycluster-auth-token-head-985vp/kube-rbac-proxy logs test.go:114: [2026-11-03T15:54:40Z] Retrieving Pod Container test-ns-rhs44/raycluster-auth-token-small-group-worker-jsd4f/ray-worker logs --- PASS: TestRayClusterAuthOptions (0.60s) === RUN TestRayClusterRHOAIImages raycluster_rhoai_images_test.go:36: no rhods-operator CSV found; skipping RHOAI image validation (standalone / PR ITS) --- SKIP: TestRayClusterRHOAIImages (0.02s) PASS ok github.com/ray-project/kuberay/ray-operator/test/e2e 109.973s === Skipped === SKIP: test/e2e TestRayClusterRHOAIImages (0.02s) raycluster_rhoai_images_test.go:36: no rhods-operator CSV found; skipping RHOAI image validation (standalone / PR ITS) DONE 36 tests, 1 skipped in 3360.140s