/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_external_oidc.py:252: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:263: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:277: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:290: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:303: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:335: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:379: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:395: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:405: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:428: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:436: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:441: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:454: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:501: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:537: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:602: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:633: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:659: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:690: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:750: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:786: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:824: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:872: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:961: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1013: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1049: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1103: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1142: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:189: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deploy/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:355: Dormant-mode test mutates controller flags; set ENABLE_TENANT_DISCOVERY_DORMANT_E2E=true/workspace/source/test/e2e/tests/test_api_keys.py:1470: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/e2e/tests/test_subscription.py:1181: gateway-only mode: per-model AuthPolicy is not created/workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners.self = <test_subscription_list_endpoints.TestListSubscriptions object at 0x7fcad8b02730> def test_model_ref_display_name_and_description_enriched(self): """Model refs include display_name and description from MaaSModelRef annotations.""" sa_name = "e2e-enrichment-sa" sa_ns = "default" maas_ns = _ns() model_ref_name = "e2e-enrichment-model-ref" model_ns = MODEL_NAMESPACE subscription_name = "e2e-enrichment-sub" auth_policy_name = "e2e-enrichment-auth" expected_display_name = "E2E Enrichment Test Model" expected_description = "Model created by e2e test to verify display_name/description enrichment" try: sa_token = _create_sa_token(sa_name, namespace=sa_ns) sa_user = _sa_to_user(sa_name, namespace=sa_ns) # Create a MaaSModelRef with display-name and description annotations. # Points to the existing e2e-distinct-simulated LLMIS so the controller # can reconcile the subscription to Active. _apply_cr({ "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSModelRef", "metadata": { "name": model_ref_name, "namespace": model_ns, "annotations": { "openshift.io/display-name": expected_display_name, "openshift.io/description": expected_description, }, }, "spec": { "modelRef": { "kind": "LLMInferenceService", "name": DISTINCT_MODEL_REF, } }, }) _create_test_subscription( subscription_name, model_ref_name, users=[sa_user], ) _create_test_auth_policy( auth_policy_name, model_ref_name, users=[sa_user], ) api_key = _create_api_key(sa_token, name=f"{sa_name}-key") > _wait_for_maas_subscription_phase(subscription_name, namespace=maas_ns) test/e2e/tests/test_subscription_list_endpoints.py:316: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-enrichment-sub', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w0-aea349', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-enrichment-sub did not reach phase 'Active' within 90s (current: phase=Degraded, modelRefStatuses=1) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_multi_tenant_integration.TestMultiTenantIntegration object at 0x7f6dae5f10d0> def test_tenant_namespace_label_change_triggers_reconciliation(self): """7.4: Adding and removing discovery labels changes reconciliation behavior.""" case = new_discovery_case(use_default_gateway=True) first_policy = case["policy_name"] second_policy = f"e2e-label-return-{case['suffix']}" try: ensure_namespace(case["tenant_ns"]) apply_tenant_cr(case["tenant_ns"], DEFAULT_GATEWAY_NAME) apply_maas_auth_policy(first_policy, case["tenant_ns"]) time.sleep(10) first = get_json_or_none("maasauthpolicy", first_policy, case["tenant_ns"]) assert first is not None assert FINALIZER_AUTHPOLICY not in ((first.get("metadata") or {}).get("finalizers") or []) apply_discovery_labels(case["tenant_ns"], case["tenant_label_name"]) wait_for_finalizer("maasauthpolicy", first_policy, case["tenant_ns"], FINALIZER_AUTHPOLICY) remove_discovery_labels(case["tenant_ns"]) time.sleep(10) > apply_maas_auth_policy(second_policy, case["tenant_ns"]) test/e2e/tests/test_multi_tenant_integration.py:220: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:855: in apply_maas_auth_policy _apply_cr( test/e2e/tests/test_helper.py:620: in _apply_cr subprocess.run(["oc", "apply", "-f", "-"], input=json.dumps(cr_dict), capture_output=True, text=True, check=True) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ input = '{"apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSAuthPolicy", "metadata": {"name": "e2e-label-return-f9151...me": "facebook-opt-125m-simulated", "namespace": "llm"}], "subjects": {"groups": [{"name": "system:authenticated"}]}}}' capture_output = True, timeout = None, check = True popenargs = (['oc', 'apply', '-f', '-'],) kwargs = {'stderr': -1, 'stdin': -1, 'stdout': -1, 'text': True} process = <Popen: returncode: 1 args: ['oc', 'apply', '-f', '-']>, stdout = '' stderr = 'error: error when retrieving current configuration of:\nResource: "maas.opendatahub.io/v1alpha1, Resource=maasauthpol...a1/namespaces/ai-tenant-e2e-mt-f9151496/maasauthpolicies/e2e-label-return-f9151496": net/http: TLS handshake timeout\n' retcode = 1 def run(*popenargs, input=None, capture_output=False, timeout=None, check=False, **kwargs): """Run command with arguments and return a CompletedProcess instance. The returned instance will have attributes args, returncode, stdout and stderr. By default, stdout and stderr are not captured, and those attributes will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them. If check is True and the exit code was non-zero, it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute, and output & stderr attributes if those streams were captured. If timeout is given, and the process takes too long, a TimeoutExpired exception will be raised. There is an optional argument "input", allowing you to pass bytes or a string to the subprocess's stdin. If you use this argument you may not also use the Popen constructor's "stdin" argument, as it will be used internally. By default, all communication is in bytes, and therefore any "input" should be bytes, and the stdout and stderr will be bytes. If in text mode, any "input" should be a string, and stdout and stderr will be strings decoded according to locale encoding, or by "encoding" if set. Text mode is triggered by setting any of text, encoding, errors or universal_newlines. The other arguments are the same as for the Popen constructor. """ if input is not None: if kwargs.get('stdin') is not None: raise ValueError('stdin and input arguments may not both be used.') kwargs['stdin'] = PIPE if capture_output: if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None: raise ValueError('stdout and stderr arguments may not be used ' 'with capture_output.') kwargs['stdout'] = PIPE kwargs['stderr'] = PIPE with Popen(*popenargs, **kwargs) as process: try: stdout, stderr = process.communicate(input, timeout=timeout) except TimeoutExpired as exc: process.kill() if _mswindows: # Windows accumulates the output in a single blocking # read() call run on child threads, with the timeout # being done in a join() on those threads. communicate() # _after_ kill() is required to collect that and add it # to the exception. exc.stdout, exc.stderr = process.communicate() else: # POSIX _communicate already populated the output so # far into the TimeoutExpired exception. process.wait() raise except: # Including KeyboardInterrupt, communicate handled that. process.kill() # We don't call process.wait() as .__exit__ does that for us. raise retcode = process.poll() if check and retcode: > raise CalledProcessError(retcode, process.args, output=stdout, stderr=stderr) E subprocess.CalledProcessError: Command '['oc', 'apply', '-f', '-']' returned non-zero exit status 1. /usr/lib64/python3.9/subprocess.py:528: CalledProcessErrorshared_test_tenants = ({'base_url': 'https://e2e-shared-a-w3-771942.apps.4fa2f5a5-8971-4ad5-8356-5e19d1253a2c.prod.konfluxeaas.com/maas-api'...me': 'e2e-shared-b-w3-231820', 'name': 'e2e-shared-b-w3-231820', 'namespace': 'ai-tenant-e2e-shared-b-w3-231820', ...}) @pytest.fixture(scope="module") def tenant_env(shared_test_tenants): """Adapter fixture with tenant-specific models.""" case_a, case_b = dict(shared_test_tenants[0]), dict(shared_test_tenants[1]) for case in (case_a, case_b): model_name = f"auth-test-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_tenant_auth_isolation.py:99: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:959: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'auth-test-model-231820' namespace = 'ai-tenant-e2e-shared-b-w3-231820' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get llminferenceservice auth-test-model-231820 -o json -n ai-tenant-e2e-shared-b-w3-231820` failed: Unable to connect to the server: net/http: TLS handshake timeout test/e2e/tests/multitenancy_helpers.py:194: RuntimeErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, ( context.distinct_model_ref, context.distinct_model_2_ref, ), ) test/e2e/tests/test_models_endpoint.py:93: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w4-5cedb0' namespace = 'e2e-models-e2e-worker-w4-5cedb0' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get llminferenceservice distinct-w4-5cedb0 -o json -n e2e-models-e2e-worker-w4-5cedb0` failed: Unable to connect to the server: net/http: TLS handshake timeout test/e2e/tests/multitenancy_helpers.py:194: RuntimeErrorshared_test_tenants = ({'base_url': 'https://e2e-shared-a-w3-771942.apps.4fa2f5a5-8971-4ad5-8356-5e19d1253a2c.prod.konfluxeaas.com/maas-api'...me': 'e2e-shared-b-w3-231820', 'name': 'e2e-shared-b-w3-231820', 'namespace': 'ai-tenant-e2e-shared-b-w3-231820', ...}) @pytest.fixture(scope="module") def tenant_env(shared_test_tenants): """Adapter fixture with tenant-specific models.""" case_a, case_b = dict(shared_test_tenants[0]), dict(shared_test_tenants[1]) for case in (case_a, case_b): model_name = f"auth-test-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_tenant_auth_isolation.py:99: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:959: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'auth-test-model-231820' namespace = 'ai-tenant-e2e-shared-b-w3-231820' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get llminferenceservice auth-test-model-231820 -o json -n ai-tenant-e2e-shared-b-w3-231820` failed: Unable to connect to the server: net/http: TLS handshake timeout test/e2e/tests/multitenancy_helpers.py:194: RuntimeErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, ( context.distinct_model_ref, context.distinct_model_2_ref, ), ) test/e2e/tests/test_models_endpoint.py:93: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w4-5cedb0' namespace = 'e2e-models-e2e-worker-w4-5cedb0' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get llminferenceservice distinct-w4-5cedb0 -o json -n e2e-models-e2e-worker-w4-5cedb0` failed: Unable to connect to the server: net/http: TLS handshake timeout test/e2e/tests/multitenancy_helpers.py:194: RuntimeErrorshared_test_tenants = ({'base_url': 'https://e2e-shared-a-w3-771942.apps.4fa2f5a5-8971-4ad5-8356-5e19d1253a2c.prod.konfluxeaas.com/maas-api'...me': 'e2e-shared-b-w3-231820', 'name': 'e2e-shared-b-w3-231820', 'namespace': 'ai-tenant-e2e-shared-b-w3-231820', ...}) @pytest.fixture(scope="module") def tenant_env(shared_test_tenants): """Adapter fixture with tenant-specific models.""" case_a, case_b = dict(shared_test_tenants[0]), dict(shared_test_tenants[1]) for case in (case_a, case_b): model_name = f"auth-test-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_tenant_auth_isolation.py:99: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:959: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'auth-test-model-231820' namespace = 'ai-tenant-e2e-shared-b-w3-231820' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get llminferenceservice auth-test-model-231820 -o json -n ai-tenant-e2e-shared-b-w3-231820` failed: Unable to connect to the server: net/http: TLS handshake timeout test/e2e/tests/multitenancy_helpers.py:194: RuntimeErrorshared_test_tenants = ({'base_url': 'https://e2e-shared-a-w3-771942.apps.4fa2f5a5-8971-4ad5-8356-5e19d1253a2c.prod.konfluxeaas.com/maas-api'...me': 'e2e-shared-b-w3-231820', 'name': 'e2e-shared-b-w3-231820', 'namespace': 'ai-tenant-e2e-shared-b-w3-231820', ...}) @pytest.fixture(scope="module") def tenant_env(shared_test_tenants): """Adapter fixture with tenant-specific models.""" case_a, case_b = dict(shared_test_tenants[0]), dict(shared_test_tenants[1]) for case in (case_a, case_b): model_name = f"auth-test-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_tenant_auth_isolation.py:99: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:959: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'auth-test-model-231820' namespace = 'ai-tenant-e2e-shared-b-w3-231820' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get llminferenceservice auth-test-model-231820 -o json -n ai-tenant-e2e-shared-b-w3-231820` failed: Unable to connect to the server: net/http: TLS handshake timeout test/e2e/tests/multitenancy_helpers.py:194: RuntimeErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, ( context.distinct_model_ref, context.distinct_model_2_ref, ), ) test/e2e/tests/test_models_endpoint.py:93: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w4-5cedb0' namespace = 'e2e-models-e2e-worker-w4-5cedb0' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get llminferenceservice distinct-w4-5cedb0 -o json -n e2e-models-e2e-worker-w4-5cedb0` failed: Unable to connect to the server: net/http: TLS handshake timeout test/e2e/tests/multitenancy_helpers.py:194: RuntimeErrorshared_test_tenants = ({'base_url': 'https://e2e-shared-a-w3-771942.apps.4fa2f5a5-8971-4ad5-8356-5e19d1253a2c.prod.konfluxeaas.com/maas-api'...me': 'e2e-shared-b-w3-231820', 'name': 'e2e-shared-b-w3-231820', 'namespace': 'ai-tenant-e2e-shared-b-w3-231820', ...}) @pytest.fixture(scope="module") def tenant_env(shared_test_tenants): """Adapter fixture with tenant-specific models.""" case_a, case_b = dict(shared_test_tenants[0]), dict(shared_test_tenants[1]) for case in (case_a, case_b): model_name = f"auth-test-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_tenant_auth_isolation.py:99: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:959: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'auth-test-model-231820' namespace = 'ai-tenant-e2e-shared-b-w3-231820' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get llminferenceservice auth-test-model-231820 -o json -n ai-tenant-e2e-shared-b-w3-231820` failed: Unable to connect to the server: net/http: TLS handshake timeout test/e2e/tests/multitenancy_helpers.py:194: RuntimeErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, ( context.distinct_model_ref, context.distinct_model_2_ref, ), ) test/e2e/tests/test_models_endpoint.py:93: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w4-5cedb0' namespace = 'e2e-models-e2e-worker-w4-5cedb0' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get llminferenceservice distinct-w4-5cedb0 -o json -n e2e-models-e2e-worker-w4-5cedb0` failed: Unable to connect to the server: net/http: TLS handshake timeout test/e2e/tests/multitenancy_helpers.py:194: RuntimeErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, ( context.distinct_model_ref, context.distinct_model_2_ref, ), ) test/e2e/tests/test_models_endpoint.py:93: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w4-5cedb0' namespace = 'e2e-models-e2e-worker-w4-5cedb0' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get llminferenceservice distinct-w4-5cedb0 -o json -n e2e-models-e2e-worker-w4-5cedb0` failed: Unable to connect to the server: net/http: TLS handshake timeout test/e2e/tests/multitenancy_helpers.py:194: RuntimeErrorself = <test_subscription_list_endpoints.TestListSubscriptionsForModel object at 0x7fcad8b02370> def test_returns_subscriptions_for_model(self): """Returns only subscriptions that include the requested model.""" sa_name = "e2e-subs-model-sa" sa_ns = "default" maas_ns = _ns() sub_with_model = "e2e-model-match-sub" sub_without_model = "e2e-model-nomatch-sub" ap_with_model = "e2e-model-match-auth" ap_without_model = "e2e-model-nomatch-auth" try: sa_token = _create_sa_token(sa_name, namespace=sa_ns) sa_user = _sa_to_user(sa_name, namespace=sa_ns) # Create two subscriptions: one with DISTINCT_MODEL_REF, one with DISTINCT_MODEL_2_REF _create_test_subscription(sub_with_model, DISTINCT_MODEL_REF, users=[sa_user]) _create_test_subscription(sub_without_model, DISTINCT_MODEL_2_REF, users=[sa_user]) _create_test_auth_policy(ap_with_model, DISTINCT_MODEL_REF, users=[sa_user]) _create_test_auth_policy(ap_without_model, DISTINCT_MODEL_2_REF, users=[sa_user]) api_key = _create_api_key(sa_token, name=f"{sa_name}-key") _wait_for_maas_subscription_phase(sub_with_model, namespace=maas_ns) _wait_for_maas_subscription_phase(sub_without_model, namespace=maas_ns) _wait_for_maas_auth_policy_phase(ap_with_model, require_enforced=False, namespace=maas_ns) _wait_for_maas_auth_policy_phase(ap_without_model, require_enforced=False, namespace=maas_ns) # Query for subscriptions that include DISTINCT_MODEL_REF url = f"{_maas_api_url()}/v1/model/{DISTINCT_MODEL_REF}/subscriptions" r = requests.get( url, headers={"Authorization": f"Bearer {api_key}"}, timeout=TIMEOUT, verify=TLS_VERIFY, ) > assert r.status_code == 200, f"Expected 200, got {r.status_code}: {r.text}" E AssertionError: Expected 200, got 500: {"error":"Exception thrown while generating token","exceptionCode":"AUTH_FAILURE","refId":"001"} E assert 500 == 200 E + where 500 = <Response [500]>.status_code test/e2e/tests/test_subscription_list_endpoints.py:407: AssertionError