/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_external_oidc.py:252: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:263: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:277: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:290: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:303: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:335: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:379: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:395: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:405: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:428: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:436: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:441: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:454: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:501: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:537: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:602: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:633: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:659: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:690: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:750: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:786: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:824: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:872: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:961: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1013: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1049: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1103: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1142: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:189: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deployself = <test_api_keys.TestAPIKeyBulkOperations object at 0x7fc421f44d30> api_keys_base_url = 'https://e2e-worker-w0-6c9f0b.apps.de6489af-f880-4a63-b730-0004b3480260.prod.konfluxeaas.com/maas-api/v1/api-keys' headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImNTakY1UUg2dTMxbnR3bTJrVkluSjhTZzNyaG83TmRJbWxKckVQWG9LUHcifQ.e...ghvKLGgNW-681roiUKvymiuB58tMqKB-ARgTgYhBRxYCX3pRpb6-ztqCVJcznXQ4Z8gV_XFkxauO_-Y2g', 'Content-Type': 'application/json'} admin_headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImNTakY1UUg2dTMxbnR3bTJrVkluSjhTZzNyaG83TmRJbWxKckVQWG9LUHcifQ.e...5gNfJ-jQWGoz6DyMZ_LEtpeyyxFCy_zDFGWahZO2mEDFDZ9rWbKMK0dRvivn0xd6HfWB8f8-KV9XRAj8g', 'Content-Type': 'application/json'} def test_bulk_revoke_dry_run_by_subscription(self, api_keys_base_url: str, headers: dict, admin_headers: dict): """Admin dry-run by subscription returns correct key count.""" if not admin_headers: pytest.skip("ADMIN_OC_TOKEN not set") sub_name = f"e2e-dry-sub-{os.urandom(4).hex()}" ns = _ns() sa_name = f"e2e-dry-sa-{os.urandom(4).hex()}" key_ids = [] try: oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) sa_headers = {"Authorization": f"Bearer {oc_token}", "Content-Type": "application/json"} _create_test_auth_policy(f"{sub_name}-auth", MODEL_REF, users=[sa_user]) _create_test_subscription(sub_name, MODEL_REF, users=[sa_user]) > _wait_for_maas_subscription_phase(sub_name, namespace=ns) test/e2e/tests/test_api_keys.py:609: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-dry-sub-19668017', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w0-6c9f0b', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-dry-sub-19668017 did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_models_endpoint.TestModelsEndpoint object at 0x7fe0cf0fb610> def test_deduplication_same_model_multiple_refs(self): """ Test 6: Same modelRef listed twice should deduplicate to 1 entry. Creates a subscription with the SAME modelRef listed TWICE (different rate limits). The API deduplicates by (model ID, URL) and returns only 1 entry since both references point to the same backend service. The response includes subscription information showing which subscription(s) provide access to the model. """ log.info("Test 6: Same modelRef twice should deduplicate (INTENDED behavior)") sa_name = "e2e-models-dedup-sa" sa_ns = "default" maas_ns = _ns() subscription_name = "e2e-dedup-subscription" auth_policy_name = "e2e-dedup-auth" api_key = None try: # Create SA with its own token sa_token = _create_sa_token(sa_name, namespace=sa_ns) sa_user = _sa_to_user(sa_name, namespace=sa_ns) # Create auth policy that grants access to the model log.info(f"Creating auth policy with access to {MODEL_REF}") auth_policy_cr = { "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSAuthPolicy", "metadata": { "name": auth_policy_name, "namespace": maas_ns, }, "spec": { "modelRefs": [{"name": MODEL_REF, "namespace": MODEL_NAMESPACE}], "subjects": { "users": [sa_user], "groups": [], }, }, } subprocess.run( ["oc", "apply", "-f", "-"], input=json.dumps(auth_policy_cr), text=True, check=True, ) # Create subscription with the SAME model ref TWICE (guaranteed duplicates) log.info(f"Creating subscription with {MODEL_REF} listed twice (to test deduplication)") subscription_cr = { "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSSubscription", "metadata": { "name": subscription_name, "namespace": maas_ns, }, "spec": { "owner": { "users": [sa_user], "groups": [], }, "modelRefs": [ { "name": MODEL_REF, "namespace": MODEL_NAMESPACE, "tokenRateLimits": [{"limit": 100, "window": "1m"}], }, { "name": MODEL_REF, # Same model ref again - guarantees duplicate "namespace": MODEL_NAMESPACE, "tokenRateLimits": [{"limit": 200, "window": "1m"}], }, ], }, } subprocess.run( ["oc", "apply", "-f", "-"], input=json.dumps(subscription_cr), text=True, check=True, ) # Wait for subscription to reconcile before creating API key > _wait_for_maas_subscription_phase(subscription_name, namespace=maas_ns) test/e2e/tests/test_models_endpoint.py:818: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-dedup-subscription', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w4-2af9a3', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-dedup-subscription did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_api_keys.TestAPIKeyBulkOperations object at 0x7fc421f44df0> api_keys_base_url = 'https://e2e-worker-w0-6c9f0b.apps.de6489af-f880-4a63-b730-0004b3480260.prod.konfluxeaas.com/maas-api/v1/api-keys' headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImNTakY1UUg2dTMxbnR3bTJrVkluSjhTZzNyaG83TmRJbWxKckVQWG9LUHcifQ.e...ghvKLGgNW-681roiUKvymiuB58tMqKB-ARgTgYhBRxYCX3pRpb6-ztqCVJcznXQ4Z8gV_XFkxauO_-Y2g', 'Content-Type': 'application/json'} admin_headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImNTakY1UUg2dTMxbnR3bTJrVkluSjhTZzNyaG83TmRJbWxKckVQWG9LUHcifQ.e...5gNfJ-jQWGoz6DyMZ_LEtpeyyxFCy_zDFGWahZO2mEDFDZ9rWbKMK0dRvivn0xd6HfWB8f8-KV9XRAj8g', 'Content-Type': 'application/json'} def test_bulk_revoke_combined_user_and_subscription(self, api_keys_base_url: str, headers: dict, admin_headers: dict): """Admin can revoke keys for a specific user within a specific subscription. Negative control: a second user's keys in the same subscription must remain active. """ if not admin_headers: pytest.skip("ADMIN_OC_TOKEN not set") sub_name = f"e2e-combo-sub-{os.urandom(4).hex()}" ns = _ns() sa_name = f"e2e-combo-sa-{os.urandom(4).hex()}" sa2_name = f"e2e-combo-sa2-{os.urandom(4).hex()}" key_ids = [] key_ids_user2 = [] try: oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) sa_headers = {"Authorization": f"Bearer {oc_token}", "Content-Type": "application/json"} oc_token2 = _create_sa_token(sa2_name, namespace=MODEL_NAMESPACE) sa_user2 = _sa_to_user(sa2_name, namespace=MODEL_NAMESPACE) sa2_headers = {"Authorization": f"Bearer {oc_token2}", "Content-Type": "application/json"} _create_test_auth_policy(f"{sub_name}-auth", MODEL_REF, users=[sa_user, sa_user2]) _create_test_subscription(sub_name, MODEL_REF, users=[sa_user, sa_user2]) > _wait_for_maas_subscription_phase(sub_name, namespace=ns) test/e2e/tests/test_api_keys.py:673: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-combo-sub-f8f3092a', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w0-6c9f0b', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-combo-sub-f8f3092a did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_models_endpoint.TestModelsEndpoint object at 0x7fe0cf0fb850> def test_multiple_distinct_models_in_subscription(self): """ Test 8: Multiple distinct models should return exactly 2 entries (1 per unique ID). Uses pre-deployed models (both known to not have backend duplication issues): - DISTINCT_MODEL_REF (simulated-distinct) serving "test/e2e-distinct-model" - DISTINCT_MODEL_2_REF (simulated-distinct-2) serving "test/e2e-distinct-model-2" Creates a subscription with both models. The API should return exactly 2 entries (one for each distinct model ID), with no duplicates. This test validates that when backend models don't have duplication bugs, the API correctly returns one entry per distinct model ID. """ log.info("Test 8: Multiple distinct models should return 2 entries") sa_name = "e2e-models-distinct-sa" sa_ns = "default" maas_ns = _ns() subscription_name = "e2e-distinct-models-subscription" auth_policy_name = "e2e-distinct-models-auth" api_key = None try: # Create SA sa_token = _create_sa_token(sa_name, namespace=sa_ns) sa_user = _sa_to_user(sa_name, namespace=sa_ns) # Create auth policy with both distinct models log.info(f"Creating auth policy with {DISTINCT_MODEL_REF} and {DISTINCT_MODEL_2_REF}") auth_policy_cr = { "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSAuthPolicy", "metadata": { "name": auth_policy_name, "namespace": maas_ns, }, "spec": { "modelRefs": [ {"name": DISTINCT_MODEL_REF, "namespace": MODEL_NAMESPACE}, {"name": DISTINCT_MODEL_2_REF, "namespace": MODEL_NAMESPACE}, ], "subjects": { "users": [sa_user], "groups": [], }, }, } subprocess.run( ["oc", "apply", "-f", "-"], input=json.dumps(auth_policy_cr), text=True, check=True, ) # Create subscription with both distinct models log.info(f"Creating subscription with {DISTINCT_MODEL_REF} and {DISTINCT_MODEL_2_REF}") subscription_cr = { "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSSubscription", "metadata": { "name": subscription_name, "namespace": maas_ns, }, "spec": { "owner": { "users": [sa_user], "groups": [], }, "modelRefs": [ { "name": DISTINCT_MODEL_REF, "namespace": MODEL_NAMESPACE, "tokenRateLimits": [{"limit": 100, "window": "1m"}], }, { "name": DISTINCT_MODEL_2_REF, "namespace": MODEL_NAMESPACE, "tokenRateLimits": [{"limit": 100, "window": "1m"}], }, ], }, } subprocess.run( ["oc", "apply", "-f", "-"], input=json.dumps(subscription_cr), text=True, check=True, ) # Wait for subscription to reconcile before creating API key > _wait_for_maas_subscription_phase(subscription_name, namespace=maas_ns) test/e2e/tests/test_models_endpoint.py:1176: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-distinct-models-subscription', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w4-2af9a3', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-distinct-models-subscription did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutError/workspace/source/test/e2e/tests/test_api_keys.py:1470: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}self = <test_api_keys.TestAPIKeySubscriptionPhases object at 0x7fc421f00a60> def test_create_key_for_active_subscription(self): """API key creation succeeds for Active subscription.""" ns = _ns() subscription_name = "e2e-apikey-active-sub" auth_name = "e2e-apikey-active-auth" sa_name = "e2e-apikey-active-sa" try: oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) _create_test_auth_policy(auth_name, MODEL_REF, users=[sa_user]) _create_test_subscription(subscription_name, MODEL_REF, users=[sa_user]) > _wait_for_maas_subscription_phase(subscription_name, namespace=ns) test/e2e/tests/test_api_keys.py:1545: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-apikey-active-sub', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w0-6c9f0b', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-apikey-active-sub did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_models_endpoint.TestModelsEndpoint object at 0x7fe0cf0fb2e0> def test_user_token_returns_all_models(self): """ Test: User token automatically returns models from all subscriptions. Creates a user with access to TWO subscriptions containing different models. Queries without X-MaaS-Subscription header and validates: - Returns models from ALL accessible subscriptions - Each model includes subscriptions array showing which subscription(s) provide access - Models appearing in multiple subscriptions have aggregated subscription list """ log.info("Test: User token returns models from all subscriptions") sa_name = "e2e-return-all-sa" sa_ns = "default" maas_ns = _ns() sub1_name = "e2e-return-all-sub1" sub2_name = "e2e-return-all-sub2" auth1_name = "e2e-return-all-auth1" auth2_name = "e2e-return-all-auth2" try: # Create SA sa_token = _create_sa_token(sa_name, namespace=sa_ns) sa_user = _sa_to_user(sa_name, namespace=sa_ns) # Create subscription 1 with DISTINCT_MODEL_REF log.info(f"Creating subscription 1 with {DISTINCT_MODEL_REF}") _create_test_auth_policy(auth1_name, DISTINCT_MODEL_REF, users=[sa_user]) _create_test_subscription(sub1_name, DISTINCT_MODEL_REF, users=[sa_user]) # Create subscription 2 with DISTINCT_MODEL_2_REF log.info(f"Creating subscription 2 with {DISTINCT_MODEL_2_REF}") _create_test_auth_policy(auth2_name, DISTINCT_MODEL_2_REF, users=[sa_user]) _create_test_subscription(sub2_name, DISTINCT_MODEL_2_REF, users=[sa_user]) _wait_for_maas_auth_policy_phase(auth1_name) _wait_for_maas_auth_policy_phase(auth2_name) > _wait_for_maas_subscription_phase(sub1_name) test/e2e/tests/test_models_endpoint.py:1283: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-return-all-sub1', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w4-2af9a3', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-return-all-sub1 did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_api_keys.TestAPIKeySubscriptionPhases object at 0x7fc421e84280> def test_create_key_for_degraded_subscription(self): """API key creation succeeds for Degraded subscription.""" ns = _ns() subscription_name = "e2e-apikey-degraded-sub" auth_name = "e2e-apikey-degraded-auth" sa_name = "e2e-apikey-degraded-sa" missing_model = "nonexistent-model-apikey" try: oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) _create_test_auth_policy(auth_name, MODEL_REF, users=[sa_user]) # Create with valid + missing model to trigger Degraded phase _create_test_subscription( subscription_name, [MODEL_REF, missing_model], users=[sa_user] ) > _wait_for_maas_subscription_phase(subscription_name, expected_phase="Degraded", namespace=ns) test/e2e/tests/test_api_keys.py:1587: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-apikey-degraded-sub', expected_phase = 'Degraded' namespace = 'ai-tenant-e2e-worker-w0-6c9f0b', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-apikey-degraded-sub did not reach phase 'Degraded' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutError/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:355: Dormant-mode test mutates controller flags; set ENABLE_TENANT_DISCOVERY_DORMANT_E2E=true/workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners.self = <test_models_endpoint.TestModelsEndpoint object at 0x7fe0cf0fb3a0> def test_user_token_with_subscription_header_filters(self): """ Test: User token with X-MaaS-Subscription header filters to that subscription. User tokens can optionally provide X-MaaS-Subscription to filter results to a specific subscription (similar to API key behavior). Expected: HTTP 200 with models from only the specified subscription. """ log.info("Test: User token with X-MaaS-Subscription header filters models") ns = _ns() auth_policy_name = "e2e-user-token-filter-auth" subscription_name = "e2e-user-token-filter-sub" sa_name = "e2e-user-token-filter-sa" try: # Create service account and token oc_token = _create_sa_token(sa_name, namespace=ns) sa_user = _sa_to_user(sa_name, namespace=ns) # Create test resources _create_test_auth_policy(auth_policy_name, MODEL_REF, users=[sa_user]) _create_test_subscription(subscription_name, MODEL_REF, users=[sa_user]) _wait_for_maas_auth_policy_phase(auth_policy_name, require_enforced=False) > _wait_for_maas_subscription_phase(subscription_name) test/e2e/tests/test_models_endpoint.py:1364: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-user-token-filter-sub', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w4-2af9a3', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-user-token-filter-sub did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutError/workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners.self = <test_multi_tenant_integration.TestMultiTenantIntegration object at 0x7fc926ffa520> def test_full_tenant_lifecycle_create_to_delete(self): """7.1: Full tenant lifecycle from create through policy/subscription reconcile to delete.""" case = new_discovery_case() role_name = f"aitenant-{case['tenant_label_name']}-tenant-admin" try: bootstrap_aitenant_tenant(case) tenant = wait_for_json(TENANT_CONFIG_KIND, TENANT_CR_NAME, case["tenant_ns"], timeout=180) tenant_labels = tenant["metadata"].get("labels") or {} tenant_annotations = tenant["metadata"].get("annotations") or {} assert tenant_labels[LABEL_MANAGED_BY_AITENANT] == "true" assert tenant_labels[LABEL_TENANT_NAME] == case["tenant_label_name"] assert tenant_labels[LABEL_TENANT_NAMESPACE] == case["tenant_ns"] assert tenant_annotations[ANNOTATION_AITENANT_NAME] == case["tenant_label_name"] assert tenant_annotations[ANNOTATION_AITENANT_NAMESPACE] == AITENANT_NAMESPACE aitenant = wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, timeout=180) assert aitenant["status"]["gatewayRef"]["name"] == case["gateway_name"] assert get_json_or_none("role", role_name, case["tenant_ns"]) is not None model_name = f"e2e-lifecycle-model-{case['suffix']}" provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) apply_maas_auth_policy( case["policy_name"], case["tenant_ns"], model_ref=model_name, model_namespace=case["tenant_ns"], ) apply_maas_subscription( case["subscription_name"], case["tenant_ns"], model_ref=model_name, model_namespace=case["tenant_ns"], ) wait_for_status_phase("maasauthpolicy", case["policy_name"], case["tenant_ns"], expected_phase="Active") > wait_for_status_phase( "maassubscription", case["subscription_name"], case["tenant_ns"], expected_phase="Active", ) test/e2e/tests/test_multi_tenant_integration.py:116: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-sub-7e508e61' namespace = 'ai-tenant-e2e-mt-7e508e61' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-sub-7e508e61 in ai-tenant-e2e-mt-7e508e61 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-sub-7e508e61","namespace":"ai-tenant-e2e-mt-7e508e61"},"spec":{"modelRefs":[{"name":"e2e-lifecycle-model-7e508e61","namespace":"ai-tenant-e2e-mt-7e508e61","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-15T13:11:23Z', 'generation': 1, 'name': 'e2e-sub-7e508e61', 'namespace': 'ai-tenant-e2e-mt-7e508e61', 'resourceVersion': '46176', 'uid': 'd54c3002-9ab1-46bc-a876-b7d32c0b3a32'}, 'spec': {'modelRefs': [{'name': 'e2e-lifecycle-model-7e508e61', 'namespace': 'ai-tenant-e2e-mt-7e508e61', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-15T13:11:23Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_api_keys.TestAPIKeySubscriptionPhases object at 0x7fc421e841f0> def test_create_key_for_failed_subscription(self): """API key creation is rejected for Failed subscription to prevent key spam.""" ns = _ns() subscription_name = "e2e-apikey-failed-sub" auth_name = "e2e-apikey-failed-auth" sa_name = "e2e-apikey-failed-sa" nonexistent_model = "nonexistent-model-apikey-failed" try: oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) _create_test_auth_policy(auth_name, MODEL_REF, users=[sa_user]) # Reference only a nonexistent model so the controller naturally # computes Failed (all model refs invalid in deriveFinalPhase). _create_test_subscription(subscription_name, nonexistent_model, users=[sa_user]) > _wait_for_maas_subscription_phase( subscription_name, expected_phase="Failed", namespace=ns ) test/e2e/tests/test_api_keys.py:1626: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-apikey-failed-sub', expected_phase = 'Failed' namespace = 'ai-tenant-e2e-worker-w0-6c9f0b', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-apikey-failed-sub did not reach phase 'Failed' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_models_endpoint.TestModelsEndpoint object at 0x7fe0cea059d0> def test_api_key_scoped_to_subscription(self): """ Test: API key returns only models from its bound subscription. API keys are scoped to a specific subscription at mint time. The gateway automatically injects X-MaaS-Subscription from the key's subscription. Expected: HTTP 200 with models only from the key's subscription, even if the user has access to multiple subscriptions. """ ns = _ns() auth_policy_name = "e2e-api-key-scoped-auth" subscription_name = "e2e-api-key-scoped-sub" sa_name = "e2e-api-key-scoped-sa" api_key = None try: # Create service account and token oc_token = _create_sa_token(sa_name, namespace=ns) sa_user = _sa_to_user(sa_name, namespace=ns) # Create test resources _create_test_auth_policy(auth_policy_name, MODEL_REF, users=[sa_user]) _create_test_subscription(subscription_name, MODEL_REF, users=[sa_user]) # Wait for subscription to reconcile before creating API key > _wait_for_maas_subscription_phase(subscription_name, namespace=ns) test/e2e/tests/test_models_endpoint.py:1601: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'e2e-api-key-scoped-sub', expected_phase = 'Active' namespace = 'ai-tenant-e2e-worker-w4-2af9a3', timeout = 90 require_model_statuses = False def _wait_for_maas_subscription_phase(name, expected_phase="Active", namespace=None, timeout=MAAS_SUBSCRIPTION_PHASE_TIMEOUT, require_model_statuses=False): """Wait for MaaSSubscription to reach a specific phase. Args: name: Name of the MaaSSubscription expected_phase: Phase to wait for (default: "Active") namespace: Namespace (defaults to _ns()) timeout: Maximum wait time in seconds (default: 60) require_model_statuses: If True, also requires modelRefStatuses to be populated (default: False). Set to True for status reporting tests. Returns: The subscription CR dict when the expected phase is reached Raises: TimeoutError: If MaaSSubscription doesn't reach expected phase within timeout """ namespace = namespace or _ns() deadline = time.time() + timeout log.info(f"Waiting for MaaSSubscription {name} to reach phase '{expected_phase}' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maassubscription", name, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") model_statuses = status.get("modelRefStatuses", []) if phase == expected_phase: if require_model_statuses: expected_count = len(cr.get("spec", {}).get("modelRefs", [])) if len(model_statuses) >= expected_count: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}' with {len(model_statuses)}/{expected_count} modelRefStatuses") return cr else: log.info(f"MaaSSubscription {name} reached phase '{expected_phase}'") return cr log.debug(f"MaaSSubscription {name}: phase={phase}, modelRefStatuses={len(model_statuses)}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maassubscription", name, namespace) status = cr.get("status", {}) if cr else {} > raise TimeoutError( f"MaaSSubscription {name} did not reach phase '{expected_phase}' within {timeout}s " f"(current: phase={status.get('phase')}, modelRefStatuses={len(status.get('modelRefStatuses', []))})" ) E TimeoutError: MaaSSubscription e2e-api-key-scoped-sub did not reach phase 'Active' within 90s (current: phase=None, modelRefStatuses=0) test/e2e/tests/test_helper.py:1334: TimeoutErrorself = <test_multi_tenant_integration.TestMultiTenantIntegration object at 0x7fc926ffadc0> def test_default_tenant_unaffected_by_multitenancy_enablement(self): """7.2: Default tenant namespace still reconciles while discovery mode is enabled.""" ns = legacy_default_namespace() suffix = uuid.uuid4().hex[:6] policy_name = f"e2e-default-int-auth-{suffix}" subscription_name = f"e2e-default-int-sub-{suffix}" try: apply_maas_auth_policy(policy_name, ns) apply_maas_subscription(subscription_name, ns) wait_for_status_phase("maasauthpolicy", policy_name, ns, expected_phase="Active") > wait_for_status_phase("maassubscription", subscription_name, ns, expected_phase=("Active", "Degraded")) test/e2e/tests/test_multi_tenant_integration.py:213: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-default-int-sub-d94bc1' namespace = 'models-as-a-service' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-default-int-sub-d94bc1 in models-as-a-service did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-default-int-sub-d94bc1","namespace":"models-as-a-service"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-15T13:15:53Z', 'generation': 1, 'name': 'e2e-default-int-sub-d94bc1', 'namespace': 'models-as-a-service', 'resourceVersion': '52038', 'uid': '90a7345b-64a1-4f0f-b17e-aecc476813ce'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-15T13:15:53Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_multi_tenant_integration.TestMultiTenantIntegration object at 0x7fc926e14e50> def test_same_named_resources_across_tenants(self): """7.3: Same-named MaaS resources in separate tenant namespaces both contribute safely.""" case_a = new_discovery_case(use_default_gateway=True) case_b = new_discovery_case(use_default_gateway=True) shared_policy = f"e2e-shared-int-policy-{case_a['suffix']}" shared_sub = f"e2e-shared-int-sub-{case_a['suffix']}" for case in (case_a, case_b): case["policy_name"] = shared_policy case["subscription_name"] = shared_sub try: for case in (case_a, case_b): apply_discovery_labels(case["tenant_ns"], case["tenant_label_name"]) apply_tenant_cr(case["tenant_ns"], DEFAULT_GATEWAY_NAME) apply_maas_auth_policy(shared_policy, case["tenant_ns"]) apply_maas_subscription(shared_sub, case["tenant_ns"]) wait_for_finalizer("maasauthpolicy", shared_policy, case["tenant_ns"], FINALIZER_AUTHPOLICY) > wait_for_finalizer("maassubscription", shared_sub, case["tenant_ns"], FINALIZER_SUBSCRIPTION) test/e2e/tests/test_multi_tenant_integration.py:235: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:264: in wait_for_finalizer return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-shared-int-sub-7b5f2fa4' namespace = 'ai-tenant-e2e-mt-7b5f2fa4' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-shared-int-sub-7b5f2fa4 in ai-tenant-e2e-mt-7b5f2fa4 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-shared-int-sub-7b5f2fa4","namespace":"ai-tenant-e2e-mt-7b5f2fa4"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-15T13:18:58Z', 'generation': 1, 'name': 'e2e-shared-int-sub-7b5f2fa4', 'namespace': 'ai-tenant-e2e-mt-7b5f2fa4', 'resourceVersion': '54359', 'uid': '84f96f42-d540-44a7-a5bd-c52089326a7d'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-15T13:18:58Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrortenant_env = ({'base_url': 'https://e2e-shared-a-w3-aa8ef1.apps.de6489af-f880-4a63-b730-0004b3480260.prod.konfluxeaas.com/maas-api'...ared-b-w3-38d90f', 'model_name': 'auth-test-model-38d90f', 'model_namespace': 'ai-tenant-e2e-shared-b-w3-38d90f', ...}) @pytest.fixture def tenant_auth_setup(tenant_env): tenant_a, tenant_b = tenant_env suffix = uuid.uuid4().hex[:6] policy_name = f"e2e-auth-iso-{suffix}" subscription_name = f"e2e-auth-iso-{suffix}" try: for tenant in tenant_env: > make_tenant_model_accessible( tenant["model_name"], tenant["namespace"], policy_name, subscription_name, gateway_name=tenant["gateway_name"], ) test/e2e/tests/test_tenant_auth_isolation.py:135: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1019: in make_tenant_model_accessible wait_for_status_phase( test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-auth-iso-8ed1c3' namespace = 'ai-tenant-e2e-shared-a-w3-aa8ef1' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-auth-iso-8ed1c3 in ai-tenant-e2e-shared-a-w3-aa8ef1 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-auth-iso-8ed1c3","namespace":"ai-tenant-e2e-shared-a-w3-aa8ef1"},"spec":{"modelRefs":[{"name":"auth-test-model-aa8ef1","namespace":"ai-tenant-e2e-shared-a-w3-aa8ef1","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-15T13:16:40Z', 'generation': 1, 'name': 'e2e-auth-iso-8ed1c3', 'namespace': 'ai-tenant-e2e-shared-a-w3-aa8ef1', 'resourceVersion': '52776', 'uid': 'd7315b90-fd53-4c6c-a749-f8cad1e498bd'}, 'spec': {'modelRefs': [{'name': 'auth-test-model-aa8ef1', 'namespace': 'ai-tenant-e2e-shared-a-w3-aa8ef1', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-15T13:16:40Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrortenant_env = ({'base_url': 'https://e2e-shared-a-w3-aa8ef1.apps.de6489af-f880-4a63-b730-0004b3480260.prod.konfluxeaas.com/maas-api'...ared-b-w3-38d90f', 'model_name': 'auth-test-model-38d90f', 'model_namespace': 'ai-tenant-e2e-shared-b-w3-38d90f', ...}) @pytest.fixture def tenant_auth_setup(tenant_env): tenant_a, tenant_b = tenant_env suffix = uuid.uuid4().hex[:6] policy_name = f"e2e-auth-iso-{suffix}" subscription_name = f"e2e-auth-iso-{suffix}" try: for tenant in tenant_env: > make_tenant_model_accessible( tenant["model_name"], tenant["namespace"], policy_name, subscription_name, gateway_name=tenant["gateway_name"], ) test/e2e/tests/test_tenant_auth_isolation.py:135: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1019: in make_tenant_model_accessible wait_for_status_phase( test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-auth-iso-909c96' namespace = 'ai-tenant-e2e-shared-a-w3-aa8ef1' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-auth-iso-909c96 in ai-tenant-e2e-shared-a-w3-aa8ef1 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-auth-iso-909c96","namespace":"ai-tenant-e2e-shared-a-w3-aa8ef1"},"spec":{"modelRefs":[{"name":"auth-test-model-aa8ef1","namespace":"ai-tenant-e2e-shared-a-w3-aa8ef1","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-15T13:19:45Z', 'generation': 1, 'name': 'e2e-auth-iso-909c96', 'namespace': 'ai-tenant-e2e-shared-a-w3-aa8ef1', 'resourceVersion': '54849', 'uid': '23287e60-572d-4747-9eb1-72da09ebb0d6'}, 'spec': {'modelRefs': [{'name': 'auth-test-model-aa8ef1', 'namespace': 'ai-tenant-e2e-shared-a-w3-aa8ef1', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-15T13:19:45Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrortenant_env = ({'base_url': 'https://e2e-shared-a-w3-aa8ef1.apps.de6489af-f880-4a63-b730-0004b3480260.prod.konfluxeaas.com/maas-api'...ared-b-w3-38d90f', 'model_name': 'auth-test-model-38d90f', 'model_namespace': 'ai-tenant-e2e-shared-b-w3-38d90f', ...}) @pytest.fixture def tenant_auth_setup(tenant_env): tenant_a, tenant_b = tenant_env suffix = uuid.uuid4().hex[:6] policy_name = f"e2e-auth-iso-{suffix}" subscription_name = f"e2e-auth-iso-{suffix}" try: for tenant in tenant_env: > make_tenant_model_accessible( tenant["model_name"], tenant["namespace"], policy_name, subscription_name, gateway_name=tenant["gateway_name"], ) test/e2e/tests/test_tenant_auth_isolation.py:135: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1019: in make_tenant_model_accessible wait_for_status_phase( test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-auth-iso-38ed44' namespace = 'ai-tenant-e2e-shared-a-w3-aa8ef1' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-auth-iso-38ed44 in ai-tenant-e2e-shared-a-w3-aa8ef1 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-auth-iso-38ed44","namespace":"ai-tenant-e2e-shared-a-w3-aa8ef1"},"spec":{"modelRefs":[{"name":"auth-test-model-aa8ef1","namespace":"ai-tenant-e2e-shared-a-w3-aa8ef1","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-15T13:22:50Z', 'generation': 1, 'name': 'e2e-auth-iso-38ed44', 'namespace': 'ai-tenant-e2e-shared-a-w3-aa8ef1', 'resourceVersion': '58455', 'uid': 'b614fe34-67c7-41b6-8b62-46afa561d371'}, 'spec': {'modelRefs': [{'name': 'auth-test-model-aa8ef1', 'namespace': 'ai-tenant-e2e-shared-a-w3-aa8ef1', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-15T13:22:50Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionError/workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:215: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_Btenant_env = ({'base_url': 'https://e2e-shared-a-w3-aa8ef1.apps.de6489af-f880-4a63-b730-0004b3480260.prod.konfluxeaas.com/maas-api'...ared-b-w3-38d90f', 'model_name': 'auth-test-model-38d90f', 'model_namespace': 'ai-tenant-e2e-shared-b-w3-38d90f', ...}) @pytest.fixture def tenant_auth_setup(tenant_env): tenant_a, tenant_b = tenant_env suffix = uuid.uuid4().hex[:6] policy_name = f"e2e-auth-iso-{suffix}" subscription_name = f"e2e-auth-iso-{suffix}" try: for tenant in tenant_env: > make_tenant_model_accessible( tenant["model_name"], tenant["namespace"], policy_name, subscription_name, gateway_name=tenant["gateway_name"], ) test/e2e/tests/test_tenant_auth_isolation.py:135: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1019: in make_tenant_model_accessible wait_for_status_phase( test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-auth-iso-4a5edc' namespace = 'ai-tenant-e2e-shared-a-w3-aa8ef1' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-auth-iso-4a5edc in ai-tenant-e2e-shared-a-w3-aa8ef1 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-auth-iso-4a5edc","namespace":"ai-tenant-e2e-shared-a-w3-aa8ef1"},"spec":{"modelRefs":[{"name":"auth-test-model-aa8ef1","namespace":"ai-tenant-e2e-shared-a-w3-aa8ef1","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-15T13:25:55Z', 'generation': 1, 'name': 'e2e-auth-iso-4a5edc', 'namespace': 'ai-tenant-e2e-shared-a-w3-aa8ef1', 'resourceVersion': '61825', 'uid': 'c709ea69-66a6-4511-a4a4-683f7dfe047e'}, 'spec': {'modelRefs': [{'name': 'auth-test-model-aa8ef1', 'namespace': 'ai-tenant-e2e-shared-a-w3-aa8ef1', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-15T13:25:55Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrortenant_env = ({'base_url': 'https://e2e-shared-a-w3-aa8ef1.apps.de6489af-f880-4a63-b730-0004b3480260.prod.konfluxeaas.com/maas-api'...ared-b-w3-38d90f', 'model_name': 'auth-test-model-38d90f', 'model_namespace': 'ai-tenant-e2e-shared-b-w3-38d90f', ...}) @pytest.fixture def tenant_auth_setup(tenant_env): tenant_a, tenant_b = tenant_env suffix = uuid.uuid4().hex[:6] policy_name = f"e2e-auth-iso-{suffix}" subscription_name = f"e2e-auth-iso-{suffix}" try: for tenant in tenant_env: > make_tenant_model_accessible( tenant["model_name"], tenant["namespace"], policy_name, subscription_name, gateway_name=tenant["gateway_name"], ) test/e2e/tests/test_tenant_auth_isolation.py:135: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1019: in make_tenant_model_accessible wait_for_status_phase( test/e2e/tests/multitenancy_helpers.py:284: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-auth-iso-18f025' namespace = 'ai-tenant-e2e-shared-a-w3-aa8ef1' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-auth-iso-18f025 in ai-tenant-e2e-shared-a-w3-aa8ef1 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-auth-iso-18f025","namespace":"ai-tenant-e2e-shared-a-w3-aa8ef1"},"spec":{"modelRefs":[{"name":"auth-test-model-aa8ef1","namespace":"ai-tenant-e2e-shared-a-w3-aa8ef1","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-09-15T13:29:01Z', 'generation': 1, 'name': 'e2e-auth-iso-18f025', 'namespace': 'ai-tenant-e2e-shared-a-w3-aa8ef1', 'resourceVersion': '63747', 'uid': '9ea338fd-ad57-4241-b7e5-d44959daa12e'}, 'spec': {'modelRefs': [{'name': 'auth-test-model-aa8ef1', 'namespace': 'ai-tenant-e2e-shared-a-w3-aa8ef1', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-15T13:29:01Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}]}} test/e2e/tests/multitenancy_helpers.py:229: AssertionError