/workspace/source/test/e2e/tests/test_namespace_scoping.py:212: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:245: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:283: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:320: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:377: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:453: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_subscription.py:1055: gateway-only mode: per-model AuthPolicy is not createdself = <test_tenant_namespace_discovery.TestTenantNamespaceDiscovery object at 0x7fbd8bbabbe0> def test_labeled_tenant_namespace_is_discovered(self): """1.1: Controller discovers labeled tenant namespaces and reconciles CRs.""" case = new_discovery_case(use_default_gateway=True) try: apply_discovery_labels(case["tenant_ns"], case["tenant_label_name"]) apply_tenant_cr(case["tenant_ns"], DEFAULT_GATEWAY_NAME) apply_maas_auth_policy(case["policy_name"], case["tenant_ns"]) apply_maas_subscription(case["subscription_name"], case["tenant_ns"]) wait_for_finalizer("maasauthpolicy", case["policy_name"], case["tenant_ns"], FINALIZER_AUTHPOLICY) wait_for_finalizer("maassubscription", case["subscription_name"], case["tenant_ns"], FINALIZER_SUBSCRIPTION) auth = wait_for_status_phase( "maasauthpolicy", case["policy_name"], case["tenant_ns"], expected_phase="Active", ) > sub = wait_for_status_phase( "maassubscription", case["subscription_name"], case["tenant_ns"], expected_phase=("Active", "Degraded"), ) test/e2e/tests/test_tenant_namespace_discovery.py:95: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:282: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maassubscription', name = 'e2e-sub-6a83eec7' namespace = 'ai-tenant-e2e-mt-6a83eec7' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maassubscription/e2e-sub-6a83eec7 in ai-tenant-e2e-mt-6a83eec7 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSSubscription', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSSubscription","metadata":{"annotations":{},"name":"e2e-sub-6a83eec7","namespace":"ai-tenant-e2e-mt-6a83eec7"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm","tokenRateLimits":[{"limit":100,"window":"1m"}]}],"owner":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-07-28T14:49:03Z', 'finalizers': ['maas.opendatahub.io/subscription-cleanup'], 'generation': 1, 'name': 'e2e-sub-6a83eec7', 'namespace': 'ai-tenant-e2e-mt-6a83eec7', 'resourceVersion': '45130', 'uid': '8a6a0304-4466-4063-a571-21605e3ccb0d'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm', 'tokenRateLimits': [{'limit': 100, 'window': '1m'}]}], 'owner': {'groups': [{'name': 'system:authenticated'}]}, 'priority': 0}, 'status': {'conditions': [{'lastTransitionTime': '2026-07-28T14:49:03Z', 'message': '', 'observedGeneration': 1, 'reason': 'NoDuplicatePeers', 'status': 'False', 'type': 'SpecPriorityDuplicate'}, {'lastTransitionTime': '2026-07-28T14:49:03Z', 'message': 'failed to reconcile TokenRateLimitPolicies: get AITenant ai-tenants/e2e-mt-6a83eec7: AITenant.maas.opendatahub.io "e2e-mt-6a83eec7" not found', 'observedGeneration': 1, 'reason': 'ReconcileFailed', 'status': 'False', 'type': 'Ready'}], 'modelRefStatuses': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm', 'ready': True, 'reason': 'Valid'}], 'phase': 'Failed'}} test/e2e/tests/multitenancy_helpers.py:227: AssertionError/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:186: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deployself = <test_tenant_namespace_discovery.TestTenantNamespaceDiscovery object at 0x7fbd8bbabe20> def test_namespace_qualified_collision_prevention(self): """1.5: Same-named CRs in two tenant namespaces use namespace-qualified TRLP tracking.""" case_a = new_discovery_case(use_default_gateway=True) case_b = new_discovery_case(use_default_gateway=True) shared_policy_name = f"e2e-shared-policy-{case_a['suffix']}" shared_sub_name = f"e2e-shared-sub-{case_a['suffix']}" case_a["policy_name"] = shared_policy_name case_a["subscription_name"] = shared_sub_name case_b["policy_name"] = shared_policy_name case_b["subscription_name"] = shared_sub_name try: for case in (case_a, case_b): apply_discovery_labels(case["tenant_ns"], case["tenant_label_name"]) apply_tenant_cr(case["tenant_ns"], DEFAULT_GATEWAY_NAME) apply_maas_auth_policy(shared_policy_name, case["tenant_ns"]) apply_maas_subscription(shared_sub_name, case["tenant_ns"]) wait_for_finalizer("maasauthpolicy", shared_policy_name, case["tenant_ns"], FINALIZER_AUTHPOLICY) wait_for_finalizer("maassubscription", shared_sub_name, case["tenant_ns"], FINALIZER_SUBSCRIPTION) _wait_for_maas_auth_policy_phase(shared_policy_name, namespace=case["tenant_ns"], timeout=120) assert_no_per_model_authpolicy(MODEL_REF, MODEL_NAMESPACE) assert get_gateway_authpolicy() is not None expected_a_sub = f"{case_a['tenant_ns']}/{shared_sub_name}" expected_b_sub = f"{case_b['tenant_ns']}/{shared_sub_name}" > sub_contributors = wait_for_annotation_contains( "tokenratelimitpolicy", f"maas-trlp-{MODEL_REF}", MODEL_NAMESPACE, "maas.opendatahub.io/subscriptions", [expected_a_sub, expected_b_sub], ) test/e2e/tests/test_tenant_namespace_discovery.py:233: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:358: in wait_for_annotation_contains wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'tokenratelimitpolicy', name = 'maas-trlp-facebook-opt-125m-simulated' namespace = 'llm' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: tokenratelimitpolicy/maas-trlp-facebook-opt-125m-simulated in llm did not satisfy condition. Last object: {'apiVersion': 'kuadrant.io/v1alpha1', 'kind': 'TokenRateLimitPolicy', 'metadata': {'annotations': {'maas.opendatahub.io/subscriptions': 'models-as-a-service/simulator-subscription'}, 'creationTimestamp': '2026-07-28T14:41:27Z', 'generation': 17, 'labels': {'app.kubernetes.io/component': 'token-rate-limit-policy', 'app.kubernetes.io/managed-by': 'maas-controller', 'app.kubernetes.io/part-of': 'maas-subscription', 'maas.opendatahub.io/model': 'facebook-opt-125m-simulated', 'maas.opendatahub.io/model-namespace': 'llm'}, 'name': 'maas-trlp-facebook-opt-125m-simulated', 'namespace': 'llm', 'ownerReferences': [{'apiVersion': 'gateway.networking.k8s.io/v1', 'blockOwnerDeletion': True, 'controller': True, 'kind': 'HTTPRoute', 'name': 'facebook-opt-125m-simulated-kserve-route', 'uid': '8fabbb8a-f9c9-443c-91da-a1da8de4a353'}], 'resourceVersion': '45319', 'uid': 'e24575f4-d282-419e-919b-485aced54ea0'}, 'spec': {'limits': {'models-as-a-service-simulator-subscription-facebook-opt-125m-simulated-tokens': {'counters': [{'expression': 'auth.identity.userid'}], 'rates': [{'limit': 100, 'window': '1m'}], 'when': [{'predicate': 'auth.identity.selected_subscription_key == "models-as-a-service/simulator-subscription@llm/facebook-opt-125m-simulated" && !request.path.endsWith("/v1/models")'}]}}, 'targetRef': {'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'facebook-opt-125m-simulated-kserve-route'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-07-28T14:41:32Z', 'message': 'TokenRateLimitPolicy has been accepted', 'reason': 'Accepted', 'status': 'True', 'type': 'Accepted'}, {'lastTransitionTime': '2026-07-28T14:49:13Z', 'message': 'TokenRateLimitPolicy has been successfully enforced', 'reason': 'Enforced', 'status': 'True', 'type': 'Enforced'}], 'observedGeneration': 17}} test/e2e/tests/multitenancy_helpers.py:227: AssertionError/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:353: Dormant-mode test mutates controller flags; set ENABLE_TENANT_DISCOVERY_DORMANT_E2E=true/workspace/source/test/e2e/tests/test_gateway_scoped_authpolicy.py:170: gateway-default-auth not present (maas-gateway-auth is active); scoping is validated by unit testsself = <test_multi_tenant_integration.TestMultiTenantIntegration object at 0x7fbd8ba59700> def test_same_named_resources_across_tenants(self): """7.3: Same-named MaaS resources in separate tenant namespaces both contribute safely.""" case_a = new_discovery_case(use_default_gateway=True) case_b = new_discovery_case(use_default_gateway=True) shared_policy = f"e2e-shared-int-policy-{case_a['suffix']}" shared_sub = f"e2e-shared-int-sub-{case_a['suffix']}" for case in (case_a, case_b): case["policy_name"] = shared_policy case["subscription_name"] = shared_sub try: for case in (case_a, case_b): apply_discovery_labels(case["tenant_ns"], case["tenant_label_name"]) apply_tenant_cr(case["tenant_ns"], DEFAULT_GATEWAY_NAME) apply_maas_auth_policy(shared_policy, case["tenant_ns"]) apply_maas_subscription(shared_sub, case["tenant_ns"]) wait_for_finalizer("maasauthpolicy", shared_policy, case["tenant_ns"], FINALIZER_AUTHPOLICY) wait_for_finalizer("maassubscription", shared_sub, case["tenant_ns"], FINALIZER_SUBSCRIPTION) wait_for_status_phase("maasauthpolicy", shared_policy, case["tenant_ns"], expected_phase="Active") expected_subs = [f"{case_a['tenant_ns']}/{shared_sub}", f"{case_b['tenant_ns']}/{shared_sub}"] > wait_for_annotation_contains( "tokenratelimitpolicy", f"maas-trlp-{MODEL_REF}", MODEL_NAMESPACE, "maas.opendatahub.io/subscriptions", expected_subs, ) test/e2e/tests/test_multi_tenant_integration.py:189: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:358: in wait_for_annotation_contains wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'tokenratelimitpolicy', name = 'maas-trlp-facebook-opt-125m-simulated' namespace = 'llm' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: tokenratelimitpolicy/maas-trlp-facebook-opt-125m-simulated in llm did not satisfy condition. Last object: {'apiVersion': 'kuadrant.io/v1alpha1', 'kind': 'TokenRateLimitPolicy', 'metadata': {'annotations': {'maas.opendatahub.io/subscriptions': 'models-as-a-service/simulator-subscription'}, 'creationTimestamp': '2026-07-28T14:41:27Z', 'generation': 19, 'labels': {'app.kubernetes.io/component': 'token-rate-limit-policy', 'app.kubernetes.io/managed-by': 'maas-controller', 'app.kubernetes.io/part-of': 'maas-subscription', 'maas.opendatahub.io/model': 'facebook-opt-125m-simulated', 'maas.opendatahub.io/model-namespace': 'llm'}, 'name': 'maas-trlp-facebook-opt-125m-simulated', 'namespace': 'llm', 'ownerReferences': [{'apiVersion': 'gateway.networking.k8s.io/v1', 'blockOwnerDeletion': True, 'controller': True, 'kind': 'HTTPRoute', 'name': 'facebook-opt-125m-simulated-kserve-route', 'uid': '8fabbb8a-f9c9-443c-91da-a1da8de4a353'}], 'resourceVersion': '56807', 'uid': 'e24575f4-d282-419e-919b-485aced54ea0'}, 'spec': {'limits': {'models-as-a-service-simulator-subscription-facebook-opt-125m-simulated-tokens': {'counters': [{'expression': 'auth.identity.userid'}], 'rates': [{'limit': 100, 'window': '1m'}], 'when': [{'predicate': 'auth.identity.selected_subscription_key == "models-as-a-service/simulator-subscription@llm/facebook-opt-125m-simulated" && !request.path.endsWith("/v1/models")'}]}}, 'targetRef': {'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'facebook-opt-125m-simulated-kserve-route'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-07-28T14:41:32Z', 'message': 'TokenRateLimitPolicy has been accepted', 'reason': 'Accepted', 'status': 'True', 'type': 'Accepted'}, {'lastTransitionTime': '2026-07-28T15:00:24Z', 'message': 'TokenRateLimitPolicy has been successfully enforced', 'reason': 'Enforced', 'status': 'True', 'type': 'Enforced'}], 'observedGeneration': 19}} test/e2e/tests/multitenancy_helpers.py:227: AssertionError/workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:212: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B/workspace/source/test/e2e/tests/test_tenant_discovery.py:134: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:210: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:131: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:179: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:266: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_external_oidc.py:249: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:260: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:274: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:287: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:300: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:332: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:376: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:392: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:402: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:425: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:433: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:438: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:451: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:498: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:534: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:599: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:630: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:656: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:687: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:747: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:771: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:824: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:872: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:926: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:978: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1014: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1068: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1107: EXTERNAL_OIDC not enabled