/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_external_oidc.py:252: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:263: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:277: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:290: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:303: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:335: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:379: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:395: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:405: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:428: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:436: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:441: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:454: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:501: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:537: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:602: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:633: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:659: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:690: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:750: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:786: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:824: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:872: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:961: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1013: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1049: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1103: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1142: EXTERNAL_OIDC not enabledself = <test_api_keys.TestAPIKeyBulkOperations object at 0x7fdd985e0b20> api_keys_base_url = 'https://e2e-worker-w0-a21a40.apps.52e29412-bb05-42e5-b2cc-67bf69d1c3bb.prod.konfluxeaas.com/maas-api/v1/api-keys' headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ikt6ZkpCSEhhWVRTV2xKaW84YUg4UnRMeTBMOHRtZzVzWXZkMV9wdWRkZDgifQ.e...Y5KS_ONTjcFGZZ61-W_4uNMnkSZopRLThw_8b3jiSZhOxzGEKRMDU5kR9JpUbxtqxBAD6uD1yEk3-u84Q', 'Content-Type': 'application/json'} admin_headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ikt6ZkpCSEhhWVRTV2xKaW84YUg4UnRMeTBMOHRtZzVzWXZkMV9wdWRkZDgifQ.e...DdJQBeF3oMYPVUiGk4OuIyr1uWdnByHAIQ2Uo_CN8y13nObIABykJ7ZCXhCkqGeDDs4tBixitlNvERb2g', 'Content-Type': 'application/json'} def test_bulk_revoke_combined_user_and_subscription(self, api_keys_base_url: str, headers: dict, admin_headers: dict): """Admin can revoke keys for a specific user within a specific subscription. Negative control: a second user's keys in the same subscription must remain active. """ if not admin_headers: pytest.skip("ADMIN_OC_TOKEN not set") sub_name = f"e2e-combo-sub-{os.urandom(4).hex()}" ns = _ns() sa_name = f"e2e-combo-sa-{os.urandom(4).hex()}" sa2_name = f"e2e-combo-sa2-{os.urandom(4).hex()}" key_ids = [] key_ids_user2 = [] try: oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) sa_headers = {"Authorization": f"Bearer {oc_token}", "Content-Type": "application/json"} oc_token2 = _create_sa_token(sa2_name, namespace=MODEL_NAMESPACE) sa_user2 = _sa_to_user(sa2_name, namespace=MODEL_NAMESPACE) sa2_headers = {"Authorization": f"Bearer {oc_token2}", "Content-Type": "application/json"} _create_test_auth_policy(f"{sub_name}-auth", MODEL_REF, users=[sa_user, sa_user2]) _create_test_subscription(sub_name, MODEL_REF, users=[sa_user, sa_user2]) > _wait_for_maas_subscription_phase(sub_name, namespace=ns) test/e2e/tests/test_api_keys.py:673: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-combo-sub-4e7e2f50', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w0-a21a40', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-combo-sub-4e7e2f50 did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutError/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:189: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deployself = <test_tenant_namespace_discovery.TestTenantNamespaceDiscovery object at 0x7f09f7fb3250> def test_namespace_qualified_collision_prevention(self): """1.5: Same-named CRs in two tenant namespaces use namespace-qualified TRLP tracking.""" case_a = new_discovery_case(use_default_gateway=True) case_b = new_discovery_case(use_default_gateway=True) shared_policy_name = f"e2e-shared-policy-{case_a['suffix']}" shared_sub_name = f"e2e-shared-sub-{case_a['suffix']}" case_a["policy_name"] = shared_policy_name case_a["subscription_name"] = shared_sub_name case_b["policy_name"] = shared_policy_name case_b["subscription_name"] = shared_sub_name try: for case in (case_a, case_b): apply_discovery_labels(case["tenant_ns"], case["tenant_label_name"]) apply_tenant_cr(case["tenant_ns"], DEFAULT_GATEWAY_NAME) apply_maas_auth_policy(shared_policy_name, case["tenant_ns"]) apply_maas_subscription(shared_sub_name, case["tenant_ns"]) wait_for_finalizer("maasauthpolicy", shared_policy_name, case["tenant_ns"], FINALIZER_AUTHPOLICY) > wait_for_finalizer("maassubscription", shared_sub_name, case["tenant_ns"], FINALIZER_SUBSCRIPTION) test/e2e/tests/test_tenant_namespace_discovery.py:228: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:264: in wait_for_finalizer return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-shared-sub-3892085f' namespace = 'ai-tenant-e2e-mt-3892085f' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-shared-sub-3892085f in ai-tenant-e2e-mt-3892085f did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-shared-sub-3892085f","namespace":"ai-tenant-e2e-mt-3892085f"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-16T13:27:19Z', 'generation': 1, 'name': 'e2e-shared-sub-3892085f', 'namespace': 'ai-tenant-e2e-mt-3892085f', 'resourceVersion': '37815', 'uid': 'ce070832-abb6-479c-a949-9148f5931524'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-16T13:27:19Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionError/workspace/source/test/e2e/tests/test_api_keys.py:1470: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}self = <test_api_keys.TestAPIKeySubscriptionPhases object at 0x7fdd9859c9a0> def test_create_key_for_active_subscription(self): """API key creation succeeds for Active subscription.""" ns = _ns() subscription_name = "e2e-apikey-active-sub" auth_name = "e2e-apikey-active-auth" sa_name = "e2e-apikey-active-sa" try: oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) _create_test_auth_policy(auth_name, MODEL_REF, users=[sa_user]) _create_test_subscription(subscription_name, MODEL_REF, users=[sa_user]) > _wait_for_maas_subscription_phase(subscription_name, namespace=ns) test/e2e/tests/test_api_keys.py:1545: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-apikey-active-sub', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w0-a21a40', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-apikey-active-sub did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_models_endpoint.TestModelsEndpoint object at 0x7fa76d5eb9a0> def test_deduplication_same_model_multiple_refs(self): """ Test 6: Same modelRef listed twice should deduplicate to 1 entry. Creates a subscription with the SAME modelRef listed TWICE (different rate limits). The API deduplicates by (model ID, URL) and returns only 1 entry since both references point to the same backend service. The response includes subscription information showing which subscription(s) provide access to the model. """ log.info("Test 6: Same modelRef twice should deduplicate (INTENDED behavior)") sa_name = "e2e-models-dedup-sa" sa_ns = "default" maas_ns = _ns() subscription_name = "e2e-dedup-subscription" auth_policy_name = "e2e-dedup-auth" api_key = None try: # Create SA with its own token sa_token = _create_sa_token(sa_name, namespace=sa_ns) sa_user = _sa_to_user(sa_name, namespace=sa_ns) # Create auth policy that grants access to the model log.info(f"Creating auth policy with access to {MODEL_REF}") auth_policy_cr = { "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSAuthPolicy", "metadata": { "name": auth_policy_name, "namespace": maas_ns, }, "spec": { "modelRefs": [{"name": MODEL_REF, "namespace": MODEL_NAMESPACE}], "subjects": { "users": [sa_user], "groups": [], }, }, } subprocess.run( ["oc", "apply", "-f", "-"], input=json.dumps(auth_policy_cr), text=True, check=True, ) # Create subscription with the SAME model ref TWICE (guaranteed duplicates) log.info(f"Creating subscription with {MODEL_REF} listed twice (to test deduplication)") subscription_cr = { "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSSubscription", "metadata": { "name": subscription_name, "namespace": maas_ns, }, "spec": { "owner": { "users": [sa_user], "groups": [], }, "modelRefs": [ { "name": MODEL_REF, "namespace": MODEL_NAMESPACE, "tokenRateLimits": [{"limit": 100, "window": "1m"}], }, { "name": MODEL_REF, # Same model ref again - guarantees duplicate "namespace": MODEL_NAMESPACE, "tokenRateLimits": [{"limit": 200, "window": "1m"}], }, ], }, } subprocess.run( ["oc", "apply", "-f", "-"], input=json.dumps(subscription_cr), text=True, check=True, ) # Wait for subscription to reconcile before creating API key > _wait_for_maas_subscription_phase(subscription_name, namespace=maas_ns) test/e2e/tests/test_models_endpoint.py:818: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-dedup-subscription', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w4-f27ea8', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-dedup-subscription did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_api_keys.TestAPIKeySubscriptionPhases object at 0x7fdd9851d400> def test_create_key_for_degraded_subscription(self): """API key creation succeeds for Degraded subscription.""" ns = _ns() subscription_name = "e2e-apikey-degraded-sub" auth_name = "e2e-apikey-degraded-auth" sa_name = "e2e-apikey-degraded-sa" missing_model = "nonexistent-model-apikey" try: oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) _create_test_auth_policy(auth_name, MODEL_REF, users=[sa_user]) # Create with valid + missing model to trigger Degraded phase _create_test_subscription( subscription_name, [MODEL_REF, missing_model], users=[sa_user] ) > _wait_for_maas_subscription_phase(subscription_name, expected_phase="Degraded", namespace=ns) test/e2e/tests/test_api_keys.py:1587: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-apikey-degraded-sub', expected_phase = 'Degraded' namespace = 'ai-tenant-e2e-worker-w0-a21a40', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-apikey-degraded-sub did not reach phase 'Degraded' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_models_endpoint.TestModelsEndpoint object at 0x7fa76d5ebbb0> def test_multiple_distinct_models_in_subscription(self): """ Test 8: Multiple distinct models should return exactly 2 entries (1 per unique ID). Uses pre-deployed models (both known to not have backend duplication issues): - DISTINCT_MODEL_REF (simulated-distinct) serving "test/e2e-distinct-model" - DISTINCT_MODEL_2_REF (simulated-distinct-2) serving "test/e2e-distinct-model-2" Creates a subscription with both models. The API should return exactly 2 entries (one for each distinct model ID), with no duplicates. This test validates that when backend models don't have duplication bugs, the API correctly returns one entry per distinct model ID. """ log.info("Test 8: Multiple distinct models should return 2 entries") sa_name = "e2e-models-distinct-sa" sa_ns = "default" maas_ns = _ns() subscription_name = "e2e-distinct-models-subscription" auth_policy_name = "e2e-distinct-models-auth" api_key = None try: # Create SA sa_token = _create_sa_token(sa_name, namespace=sa_ns) sa_user = _sa_to_user(sa_name, namespace=sa_ns) # Create auth policy with both distinct models log.info(f"Creating auth policy with {DISTINCT_MODEL_REF} and {DISTINCT_MODEL_2_REF}") auth_policy_cr = { "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSAuthPolicy", "metadata": { "name": auth_policy_name, "namespace": maas_ns, }, "spec": { "modelRefs": [ {"name": DISTINCT_MODEL_REF, "namespace": MODEL_NAMESPACE}, {"name": DISTINCT_MODEL_2_REF, "namespace": MODEL_NAMESPACE}, ], "subjects": { "users": [sa_user], "groups": [], }, }, } subprocess.run( ["oc", "apply", "-f", "-"], input=json.dumps(auth_policy_cr), text=True, check=True, ) # Create subscription with both distinct models log.info(f"Creating subscription with {DISTINCT_MODEL_REF} and {DISTINCT_MODEL_2_REF}") subscription_cr = { "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSSubscription", "metadata": { "name": subscription_name, "namespace": maas_ns, }, "spec": { "owner": { "users": [sa_user], "groups": [], }, "modelRefs": [ { "name": DISTINCT_MODEL_REF, "namespace": MODEL_NAMESPACE, "tokenRateLimits": [{"limit": 100, "window": "1m"}], }, { "name": DISTINCT_MODEL_2_REF, "namespace": MODEL_NAMESPACE, "tokenRateLimits": [{"limit": 100, "window": "1m"}], }, ], }, } subprocess.run( ["oc", "apply", "-f", "-"], input=json.dumps(subscription_cr), text=True, check=True, ) # Wait for subscription to reconcile before creating API key > _wait_for_maas_subscription_phase(subscription_name, namespace=maas_ns) test/e2e/tests/test_models_endpoint.py:1176: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-distinct-models-subscription', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w4-f27ea8', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-distinct-models-subscription did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutError/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:355: Dormant-mode test mutates controller flags; set ENABLE_TENANT_DISCOVERY_DORMANT_E2E=trueself = <test_api_keys.TestAPIKeySubscriptionPhases object at 0x7fdd9851d370> def test_create_key_for_failed_subscription(self): """API key creation is rejected for Failed subscription to prevent key spam.""" ns = _ns() subscription_name = "e2e-apikey-failed-sub" auth_name = "e2e-apikey-failed-auth" sa_name = "e2e-apikey-failed-sa" nonexistent_model = "nonexistent-model-apikey-failed" try: oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) _create_test_auth_policy(auth_name, MODEL_REF, users=[sa_user]) # Reference only a nonexistent model so the controller naturally # computes Failed (all model refs invalid in deriveFinalPhase). _create_test_subscription(subscription_name, nonexistent_model, users=[sa_user]) > _wait_for_maas_subscription_phase( subscription_name, expected_phase="Failed", namespace=ns ) test/e2e/tests/test_api_keys.py:1626: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-apikey-failed-sub', expected_phase = 'Failed' namespace = 'ai-tenant-e2e-worker-w0-a21a40', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-apikey-failed-sub did not reach phase 'Failed' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_models_endpoint.TestModelsEndpoint object at 0x7fa76d5eb310> def test_user_token_returns_all_models(self): """ Test: User token automatically returns models from all subscriptions. Creates a user with access to TWO subscriptions containing different models. Queries without X-MaaS-Subscription header and validates: - Returns models from ALL accessible subscriptions - Each model includes subscriptions array showing which subscription(s) provide access - Models appearing in multiple subscriptions have aggregated subscription list """ log.info("Test: User token returns models from all subscriptions") sa_name = "e2e-return-all-sa" sa_ns = "default" maas_ns = _ns() sub1_name = "e2e-return-all-sub1" sub2_name = "e2e-return-all-sub2" auth1_name = "e2e-return-all-auth1" auth2_name = "e2e-return-all-auth2" try: # Create SA sa_token = _create_sa_token(sa_name, namespace=sa_ns) sa_user = _sa_to_user(sa_name, namespace=sa_ns) # Create subscription 1 with DISTINCT_MODEL_REF log.info(f"Creating subscription 1 with {DISTINCT_MODEL_REF}") _create_test_auth_policy(auth1_name, DISTINCT_MODEL_REF, users=[sa_user]) _create_test_subscription(sub1_name, DISTINCT_MODEL_REF, users=[sa_user]) # Create subscription 2 with DISTINCT_MODEL_2_REF log.info(f"Creating subscription 2 with {DISTINCT_MODEL_2_REF}") _create_test_auth_policy(auth2_name, DISTINCT_MODEL_2_REF, users=[sa_user]) _create_test_subscription(sub2_name, DISTINCT_MODEL_2_REF, users=[sa_user]) _wait_for_maas_auth_policy_phase(auth1_name) _wait_for_maas_auth_policy_phase(auth2_name) > _wait_for_maas_subscription_phase(sub1_name) test/e2e/tests/test_models_endpoint.py:1283: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-return-all-sub1', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w4-f27ea8', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-return-all-sub1 did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutError/workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners.self = <test_multi_tenant_integration.TestMultiTenantIntegration object at 0x7f09fc5ddfd0> def test_full_tenant_lifecycle_create_to_delete(self): """7.1: Full tenant lifecycle from create through policy/subscription reconcile to delete.""" case = new_discovery_case() role_name = f"aitenant-{case['tenant_label_name']}-tenant-admin" try: bootstrap_aitenant_tenant(case) tenant = wait_for_json(TENANT_CONFIG_KIND, TENANT_CR_NAME, case["tenant_ns"], timeout=180) tenant_labels = tenant["metadata"].get("labels") or {} tenant_annotations = tenant["metadata"].get("annotations") or {} assert tenant_labels[LABEL_MANAGED_BY_AITENANT] == "true" assert tenant_labels[LABEL_TENANT_NAME] == case["tenant_label_name"] assert tenant_labels[LABEL_TENANT_NAMESPACE] == case["tenant_ns"] assert tenant_annotations[ANNOTATION_AITENANT_NAME] == case["tenant_label_name"] assert tenant_annotations[ANNOTATION_AITENANT_NAMESPACE] == AITENANT_NAMESPACE aitenant = wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, timeout=180) assert aitenant["status"]["gatewayRef"]["name"] == case["gateway_name"] assert get_json_or_none("role", role_name, case["tenant_ns"]) is not None model_name = f"e2e-lifecycle-model-{case['suffix']}" provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) apply_maas_auth_policy( case["policy_name"], case["tenant_ns"], model_ref=model_name, model_namespace=case["tenant_ns"], ) apply_maas_subscription( case["subscription_name"], case["tenant_ns"], model_ref=model_name, model_namespace=case["tenant_ns"], ) wait_for_status_phase("maasauthpolicy", case["policy_name"], case["tenant_ns"], expected_phase="Active") > wait_for_status_phase( "maassubscription", case["subscription_name"], case["tenant_ns"], expected_phase="Active", ) test/e2e/tests/test_multi_tenant_integration.py:116: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-sub-4a235ace' namespace = 'ai-tenant-e2e-mt-4a235ace' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-sub-4a235ace in ai-tenant-e2e-mt-4a235ace did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-sub-4a235ace","namespace":"ai-tenant-e2e-mt-4a235ace"},"spec":{"modelRefs":[{"name":"e2e-lifecycle-model-4a235ace","namespace":"ai-tenant-e2e-mt-4a235ace","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-16T13:33:02Z', 'generation': 1, 'name': 'e2e-sub-4a235ace', 'namespace': 'ai-tenant-e2e-mt-4a235ace', 'resourceVersion': '45430', 'uid': 'af869355-4038-4608-9a48-ea3579c58830'}, 'spec': {'modelRefs': [{'name': 'e2e-lifecycle-model-4a235ace', 'namespace': 'ai-tenant-e2e-mt-4a235ace', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-16T13:33:02Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_api_keys.TestAPIKeySubscriptionFilter object at 0x7fdd9851de80> api_keys_base_url = 'https://e2e-worker-w0-a21a40.apps.52e29412-bb05-42e5-b2cc-67bf69d1c3bb.prod.konfluxeaas.com/maas-api/v1/api-keys' headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ikt6ZkpCSEhhWVRTV2xKaW84YUg4UnRMeTBMOHRtZzVzWXZkMV9wdWRkZDgifQ.e...Y5KS_ONTjcFGZZ61-W_4uNMnkSZopRLThw_8b3jiSZhOxzGEKRMDU5kR9JpUbxtqxBAD6uD1yEk3-u84Q', 'Content-Type': 'application/json'} def test_search_filters_by_subscription(self, api_keys_base_url: str, headers: dict): """Search with subscription filter returns only keys bound to that subscription.""" sub_a = f"e2e-filter-sub-a-{os.urandom(4).hex()}" sub_b = f"e2e-filter-sub-b-{os.urandom(4).hex()}" ns = _ns() sa_name = f"e2e-filter-sa-{os.urandom(4).hex()}" key_ids_a = [] key_ids_b = [] try: # Create one SA authorized for both subscriptions so that # exclusion in search results is attributable to the subscription # filter, not user-scoping. oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) sa_headers = {"Authorization": f"Bearer {oc_token}", "Content-Type": "application/json"} _create_test_auth_policy(f"{sub_a}-auth", MODEL_REF, users=[sa_user]) _create_test_subscription(sub_a, MODEL_REF, users=[sa_user]) > _wait_for_maas_subscription_phase(sub_a, namespace=ns) test/e2e/tests/test_api_keys.py:1873: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-filter-sub-a-01844d7b', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w0-a21a40', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-filter-sub-a-01844d7b did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_models_endpoint.TestModelsEndpoint object at 0x7fa76d6735b0> def test_user_token_with_subscription_header_filters(self): """ Test: User token with X-MaaS-Subscription header filters to that subscription. User tokens can optionally provide X-MaaS-Subscription to filter results to a specific subscription (similar to API key behavior). Expected: HTTP 200 with models from only the specified subscription. """ log.info("Test: User token with X-MaaS-Subscription header filters models") ns = _ns() auth_policy_name = "e2e-user-token-filter-auth" subscription_name = "e2e-user-token-filter-sub" sa_name = "e2e-user-token-filter-sa" try: # Create service account and token oc_token = _create_sa_token(sa_name, namespace=ns) sa_user = _sa_to_user(sa_name, namespace=ns) # Create test resources _create_test_auth_policy(auth_policy_name, MODEL_REF, users=[sa_user]) _create_test_subscription(subscription_name, MODEL_REF, users=[sa_user]) _wait_for_maas_auth_policy_phase(auth_policy_name, require_enforced=False) > _wait_for_maas_subscription_phase(subscription_name) test/e2e/tests/test_models_endpoint.py:1364: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-user-token-filter-sub', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w4-f27ea8', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-user-token-filter-sub did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_models_endpoint.TestModelsEndpoint object at 0x7fa76d673b20> def test_api_key_scoped_to_subscription(self): """ Test: API key returns only models from its bound subscription. API keys are scoped to a specific subscription at mint time. The gateway automatically injects X-MaaS-Subscription from the key's subscription. Expected: HTTP 200 with models only from the key's subscription, even if the user has access to multiple subscriptions. """ ns = _ns() auth_policy_name = "e2e-api-key-scoped-auth" subscription_name = "e2e-api-key-scoped-sub" sa_name = "e2e-api-key-scoped-sa" api_key = None try: # Create service account and token oc_token = _create_sa_token(sa_name, namespace=ns) sa_user = _sa_to_user(sa_name, namespace=ns) # Create test resources _create_test_auth_policy(auth_policy_name, MODEL_REF, users=[sa_user]) _create_test_subscription(subscription_name, MODEL_REF, users=[sa_user]) # Wait for subscription to reconcile before creating API key > _wait_for_maas_subscription_phase(subscription_name, namespace=ns) test/e2e/tests/test_models_endpoint.py:1601: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-api-key-scoped-sub', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w4-f27ea8', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-api-key-scoped-sub did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_multi_tenant_integration.TestMultiTenantIntegration object at 0x7f09fc065970> def test_default_tenant_unaffected_by_multitenancy_enablement(self): """7.2: Default tenant namespace still reconciles while discovery mode is enabled.""" ns = legacy_default_namespace() suffix = uuid.uuid4().hex[:6] policy_name = f"e2e-default-int-auth-{suffix}" subscription_name = f"e2e-default-int-sub-{suffix}" try: apply_maas_auth_policy(policy_name, ns) apply_maas_subscription(subscription_name, ns) wait_for_status_phase("maasauthpolicy", policy_name, ns, expected_phase="Active") > wait_for_status_phase("maassubscription", subscription_name, ns, expected_phase=("Active", "Degraded")) test/e2e/tests/test_multi_tenant_integration.py:213: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-default-int-sub-e5ef3b' namespace = 'models-as-a-service' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-default-int-sub-e5ef3b in models-as-a-service did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-default-int-sub-e5ef3b","namespace":"models-as-a-service"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-16T13:37:27Z', 'generation': 1, 'name': 'e2e-default-int-sub-e5ef3b', 'namespace': 'models-as-a-service', 'resourceVersion': '50016', 'uid': '791c155e-8480-44fe-b331-62db4fad2a78'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-16T13:37:27Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_multi_tenant_integration.TestMultiTenantIntegration object at 0x7f09fc065bb0> def test_same_named_resources_across_tenants(self): """7.3: Same-named MaaS resources in separate tenant namespaces both contribute safely.""" case_a = new_discovery_case(use_default_gateway=True) case_b = new_discovery_case(use_default_gateway=True) shared_policy = f"e2e-shared-int-policy-{case_a['suffix']}" shared_sub = f"e2e-shared-int-sub-{case_a['suffix']}" for case in (case_a, case_b): case["policy_name"] = shared_policy case["subscription_name"] = shared_sub try: for case in (case_a, case_b): apply_discovery_labels(case["tenant_ns"], case["tenant_label_name"]) apply_tenant_cr(case["tenant_ns"], DEFAULT_GATEWAY_NAME) apply_maas_auth_policy(shared_policy, case["tenant_ns"]) apply_maas_subscription(shared_sub, case["tenant_ns"]) wait_for_finalizer("maasauthpolicy", shared_policy, case["tenant_ns"], FINALIZER_AUTHPOLICY) > wait_for_finalizer("maassubscription", shared_sub, case["tenant_ns"], FINALIZER_SUBSCRIPTION) test/e2e/tests/test_multi_tenant_integration.py:235: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:264: in wait_for_finalizer return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-shared-int-sub-55cf9cd6' namespace = 'ai-tenant-e2e-mt-55cf9cd6' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-shared-int-sub-55cf9cd6 in ai-tenant-e2e-mt-55cf9cd6 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-shared-int-sub-55cf9cd6","namespace":"ai-tenant-e2e-mt-55cf9cd6"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-16T13:40:32Z', 'generation': 1, 'name': 'e2e-shared-int-sub-55cf9cd6', 'namespace': 'ai-tenant-e2e-mt-55cf9cd6', 'resourceVersion': '53285', 'uid': '82a5b478-97e6-40d6-af0f-9fefefced868'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-16T13:40:32Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrortenant_env = ({'base_url': 'https://e2e-shared-a-w3-7e7bb4.apps.52e29412-bb05-42e5-b2cc-67bf69d1c3bb.prod.konfluxeaas.com/maas-api'...ared-b-w3-8c3cdf', 'model_name': 'auth-test-model-8c3cdf', 'model_namespace': 'ai-tenant-e2e-shared-b-w3-8c3cdf', ...}) @pytest.fixture def tenant_auth_setup(tenant_env): tenant_a, tenant_b = tenant_env suffix = uuid.uuid4().hex[:6] policy_name = f"e2e-auth-iso-{suffix}" subscription_name = f"e2e-auth-iso-{suffix}" try: for tenant in tenant_env: > make_tenant_model_accessible( tenant["model_name"], tenant["namespace"], policy_name, subscription_name, gateway_name=tenant["gateway_name"], ) test/e2e/tests/test_tenant_auth_isolation.py:135: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1019: in make_tenant_model_accessible wait_for_status_phase( test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-auth-iso-a156be' namespace = 'ai-tenant-e2e-shared-a-w3-7e7bb4' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-auth-iso-a156be in ai-tenant-e2e-shared-a-w3-7e7bb4 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-auth-iso-a156be","namespace":"ai-tenant-e2e-shared-a-w3-7e7bb4"},"spec":{"modelRefs":[{"name":"auth-test-model-7e7bb4","namespace":"ai-tenant-e2e-shared-a-w3-7e7bb4","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-16T13:38:48Z', 'generation': 1, 'name': 'e2e-auth-iso-a156be', 'namespace': 'ai-tenant-e2e-shared-a-w3-7e7bb4', 'resourceVersion': '51799', 'uid': '1a367e9e-f8c7-4f93-8544-35ff2ebfad41'}, 'spec': {'modelRefs': [{'name': 'auth-test-model-7e7bb4', 'namespace': 'ai-tenant-e2e-shared-a-w3-7e7bb4', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-16T13:38:48Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrortenant_env = ({'base_url': 'https://e2e-shared-a-w3-7e7bb4.apps.52e29412-bb05-42e5-b2cc-67bf69d1c3bb.prod.konfluxeaas.com/maas-api'...ared-b-w3-8c3cdf', 'model_name': 'auth-test-model-8c3cdf', 'model_namespace': 'ai-tenant-e2e-shared-b-w3-8c3cdf', ...}) @pytest.fixture def tenant_auth_setup(tenant_env): tenant_a, tenant_b = tenant_env suffix = uuid.uuid4().hex[:6] policy_name = f"e2e-auth-iso-{suffix}" subscription_name = f"e2e-auth-iso-{suffix}" try: for tenant in tenant_env: > make_tenant_model_accessible( tenant["model_name"], tenant["namespace"], policy_name, subscription_name, gateway_name=tenant["gateway_name"], ) test/e2e/tests/test_tenant_auth_isolation.py:135: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1019: in make_tenant_model_accessible wait_for_status_phase( test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-auth-iso-31353a' namespace = 'ai-tenant-e2e-shared-a-w3-7e7bb4' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-auth-iso-31353a in ai-tenant-e2e-shared-a-w3-7e7bb4 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-auth-iso-31353a","namespace":"ai-tenant-e2e-shared-a-w3-7e7bb4"},"spec":{"modelRefs":[{"name":"auth-test-model-7e7bb4","namespace":"ai-tenant-e2e-shared-a-w3-7e7bb4","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-16T13:41:53Z', 'generation': 1, 'name': 'e2e-auth-iso-31353a', 'namespace': 'ai-tenant-e2e-shared-a-w3-7e7bb4', 'resourceVersion': '54223', 'uid': '239e8059-3bf6-4502-b264-e7ca34238ea2'}, 'spec': {'modelRefs': [{'name': 'auth-test-model-7e7bb4', 'namespace': 'ai-tenant-e2e-shared-a-w3-7e7bb4', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-16T13:41:53Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrortenant_env = ({'base_url': 'https://e2e-shared-a-w3-7e7bb4.apps.52e29412-bb05-42e5-b2cc-67bf69d1c3bb.prod.konfluxeaas.com/maas-api'...ared-b-w3-8c3cdf', 'model_name': 'auth-test-model-8c3cdf', 'model_namespace': 'ai-tenant-e2e-shared-b-w3-8c3cdf', ...}) @pytest.fixture def tenant_auth_setup(tenant_env): tenant_a, tenant_b = tenant_env suffix = uuid.uuid4().hex[:6] policy_name = f"e2e-auth-iso-{suffix}" subscription_name = f"e2e-auth-iso-{suffix}" try: for tenant in tenant_env: > make_tenant_model_accessible( tenant["model_name"], tenant["namespace"], policy_name, subscription_name, gateway_name=tenant["gateway_name"], ) test/e2e/tests/test_tenant_auth_isolation.py:135: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1019: in make_tenant_model_accessible wait_for_status_phase( test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-auth-iso-1d443d' namespace = 'ai-tenant-e2e-shared-a-w3-7e7bb4' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-auth-iso-1d443d in ai-tenant-e2e-shared-a-w3-7e7bb4 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-auth-iso-1d443d","namespace":"ai-tenant-e2e-shared-a-w3-7e7bb4"},"spec":{"modelRefs":[{"name":"auth-test-model-7e7bb4","namespace":"ai-tenant-e2e-shared-a-w3-7e7bb4","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-16T13:45:00Z', 'generation': 1, 'name': 'e2e-auth-iso-1d443d', 'namespace': 'ai-tenant-e2e-shared-a-w3-7e7bb4', 'resourceVersion': '58227', 'uid': '88cc31c7-a385-4545-b914-6139855bc42c'}, 'spec': {'modelRefs': [{'name': 'auth-test-model-7e7bb4', 'namespace': 'ai-tenant-e2e-shared-a-w3-7e7bb4', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-16T13:45:00Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionError/workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:215: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_Btenant_env = ({'base_url': 'https://e2e-shared-a-w3-7e7bb4.apps.52e29412-bb05-42e5-b2cc-67bf69d1c3bb.prod.konfluxeaas.com/maas-api'...ared-b-w3-8c3cdf', 'model_name': 'auth-test-model-8c3cdf', 'model_namespace': 'ai-tenant-e2e-shared-b-w3-8c3cdf', ...}) @pytest.fixture def tenant_auth_setup(tenant_env): tenant_a, tenant_b = tenant_env suffix = uuid.uuid4().hex[:6] policy_name = f"e2e-auth-iso-{suffix}" subscription_name = f"e2e-auth-iso-{suffix}" try: for tenant in tenant_env: > make_tenant_model_accessible( tenant["model_name"], tenant["namespace"], policy_name, subscription_name, gateway_name=tenant["gateway_name"], ) test/e2e/tests/test_tenant_auth_isolation.py:135: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1019: in make_tenant_model_accessible wait_for_status_phase( test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-auth-iso-3ea83d' namespace = 'ai-tenant-e2e-shared-a-w3-7e7bb4' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-auth-iso-3ea83d in ai-tenant-e2e-shared-a-w3-7e7bb4 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-auth-iso-3ea83d","namespace":"ai-tenant-e2e-shared-a-w3-7e7bb4"},"spec":{"modelRefs":[{"name":"auth-test-model-7e7bb4","namespace":"ai-tenant-e2e-shared-a-w3-7e7bb4","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-16T13:48:05Z', 'generation': 1, 'name': 'e2e-auth-iso-3ea83d', 'namespace': 'ai-tenant-e2e-shared-a-w3-7e7bb4', 'resourceVersion': '61403', 'uid': '913d983a-c279-412a-bc48-ac63d947ed1c'}, 'spec': {'modelRefs': [{'name': 'auth-test-model-7e7bb4', 'namespace': 'ai-tenant-e2e-shared-a-w3-7e7bb4', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-16T13:48:06Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrortenant_env = ({'base_url': 'https://e2e-shared-a-w3-7e7bb4.apps.52e29412-bb05-42e5-b2cc-67bf69d1c3bb.prod.konfluxeaas.com/maas-api'...ared-b-w3-8c3cdf', 'model_name': 'auth-test-model-8c3cdf', 'model_namespace': 'ai-tenant-e2e-shared-b-w3-8c3cdf', ...}) @pytest.fixture def tenant_auth_setup(tenant_env): tenant_a, tenant_b = tenant_env suffix = uuid.uuid4().hex[:6] policy_name = f"e2e-auth-iso-{suffix}" subscription_name = f"e2e-auth-iso-{suffix}" try: for tenant in tenant_env: > make_tenant_model_accessible( tenant["model_name"], tenant["namespace"], policy_name, subscription_name, gateway_name=tenant["gateway_name"], ) test/e2e/tests/test_tenant_auth_isolation.py:135: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1019: in make_tenant_model_accessible wait_for_status_phase( test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-auth-iso-fe18db' namespace = 'ai-tenant-e2e-shared-a-w3-7e7bb4' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-auth-iso-fe18db in ai-tenant-e2e-shared-a-w3-7e7bb4 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-auth-iso-fe18db","namespace":"ai-tenant-e2e-shared-a-w3-7e7bb4"},"spec":{"modelRefs":[{"name":"auth-test-model-7e7bb4","namespace":"ai-tenant-e2e-shared-a-w3-7e7bb4","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-16T13:51:11Z', 'generation': 1, 'name': 'e2e-auth-iso-fe18db', 'namespace': 'ai-tenant-e2e-shared-a-w3-7e7bb4', 'resourceVersion': '63322', 'uid': 'ff91e7cf-3184-457d-b7f8-25a6ba704f78'}, 'spec': {'modelRefs': [{'name': 'auth-test-model-7e7bb4', 'namespace': 'ai-tenant-e2e-shared-a-w3-7e7bb4', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-16T13:51:11Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionError