/workspace/source/test/maas-e2e/test/e2e/tests/test_api_keys.py:1429: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners.api_keys_base_url = 'https://maas.apps.c4b112aa94bb28591ff4.prod.konflux-ocp-ci.dev/maas-api/v1/api-keys' headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImQzVHE2ekloQVJGWGpjSEwwTEdORkh5emRtQmdOaGJkTkZrZkEtdWV0ODQifQ.e...Y6WT5rt4dQAHhsaF-cTWFbRlyZS33hw9294vISUw2hhEI77c1UpMObA057Uazl1HaYn-iK1g5nm1ohhbw', 'Content-Type': 'application/json'} @pytest.fixture(scope="module") def api_key(api_keys_base_url: str, headers: dict) -> str: """ Create an API key for model inference tests. Returns the plaintext API key (show-once pattern). Note: The key inherits the authenticated user's groups, which should include system:authenticated to satisfy AuthPolicy requirements for model access. """ from multitenancy_helpers import response_summary sim_sub = os.environ.get("E2E_SIMULATOR_SUBSCRIPTION", "simulator-subscription") key_name = f"e2e-test-inference-key-{_xdist_worker_suffix()}" print(f"[api_key] Creating API key for inference tests ({key_name})...") r = requests.post( api_keys_base_url, headers=headers, json={"name": key_name, "subscription": sim_sub}, timeout=30, verify=TLS_VERIFY, ) # Accept both 200 and 201 as success if r.status_code not in (200, 201): > raise RuntimeError(f"Failed to create API key: {response_summary(r)}") E RuntimeError: Failed to create API key: status=500 x-ext-auth-reason=None body=Internal Server Error. test/maas-e2e/test/e2e/tests/conftest.py:385: RuntimeError/workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py:190: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deploy/workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py:212: legacy gateway-default-auth is scoped to the shared default gateway/workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners.@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-925e04-auth' namespace = 'ai-tenant-e2e-inf-a-925e04' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f59f8d67880> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-925e04-auth in ai-tenant-e2e-inf-a-925e04 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-925e04-auth","namespace":"ai-tenant-e2e-inf-a-925e04"},"spec":{"modelRefs":[{"name":"test-model-925e04","namespace":"ai-tenant-e2e-inf-a-925e04"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:06:27Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-925e04-auth', 'namespace': 'ai-tenant-e2e-inf-a-925e04', 'resourceVersion': '91617', 'uid': 'e24a9d2c-e90e-472c-8d3b-5a3a841b5f5b'}, 'spec': {'modelRefs': [{'name': 'test-model-925e04', 'namespace': 'ai-tenant-e2e-inf-a-925e04'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:06:27Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-a-925e04-maas-auth to be accepted and enforced: AuthPolicy waiting for the following components to sync: [AuthConfig (0dfb279c544b51930363b74f9c094bd5282b8605d5b89971b817ad7ac67e95d7) AuthConfig (a474acfff2565f8b107c128bac895755b62fadc04d8c5361cfb26982839655e4) AuthConfig (9ca9d36d05295a7fe149c18cf778c8af3f3641ba59bfc952c2ae8ced324dd54b) AuthConfig (df67becca0f498b8b15358f6c309454d1c6e4309ffb989a85e75c2a5f997cb73) AuthConfig (1e4d690e0f7e5fdcec40cb088afd48595372c059365c5ab6d40f7bf998868f28) AuthConfig (55c88eaefe365547076fd1cfbfba73a8d08daf5333d2e7bc2a260d6406e7b0c8) AuthConfig (0e2761ca506777a19c32a3aa3a9ac71cd63ec7d9c4469a08dfcf53d45a726fb7) AuthConfig (3b2e3304ad0728c8197de416386777cc9d9520c00cebee3b46158713d607462a) AuthConfig (adfe8ec00d6b3d008c5b6935d19959fba7a2d27e5479f5697f8f044971635e95) AuthConfig (4e1c58168878c58f2246a1027d2f36eca0e8dfa22e022cb13ca2131317e7b205) AuthConfig (d0a5e1c4fd704f648d62c7bcfcc6d2a220bc153f841cc033af0f2a94fcc13e85) AuthConfig (609f5ca65ad4b48bba155787a66b474a2424a038c1efb7308c84d6aec86e5ee3) AuthConfig (e0e1ef48215cb0bb0ed010f3821057593448dc2784fbdd0e047dfdee508ca5ea) AuthConfig (d467fc629c3032d2e3876533570324782c2ad0beb8ba75f3dad6095923c15ea2) AuthConfig (b78c3a047d12038327f2fcdd35049bc882d96cf4704bc8dd73300811f830a1d9) AuthConfig (1d80346f8d05263593f54ddbe5d75f66b16002bc3e3bf62d4f6edddb737fc1b2) AuthConfig (b9da8a94e1df259b48c5017f1de85eccbfc81822d7b665efd86197f20b97e4b6) AuthConfig (8f2faa6bd4b4b5b190a10eace452e1efeb59269b176db8ad8013495275eb7c54)]', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-925e04-auth' namespace = 'ai-tenant-e2e-inf-a-925e04' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f59f8d67880> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-925e04-auth in ai-tenant-e2e-inf-a-925e04 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-925e04-auth","namespace":"ai-tenant-e2e-inf-a-925e04"},"spec":{"modelRefs":[{"name":"test-model-925e04","namespace":"ai-tenant-e2e-inf-a-925e04"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:06:27Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-925e04-auth', 'namespace': 'ai-tenant-e2e-inf-a-925e04', 'resourceVersion': '91617', 'uid': 'e24a9d2c-e90e-472c-8d3b-5a3a841b5f5b'}, 'spec': {'modelRefs': [{'name': 'test-model-925e04', 'namespace': 'ai-tenant-e2e-inf-a-925e04'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:06:27Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-a-925e04-maas-auth to be accepted and enforced: AuthPolicy waiting for the following components to sync: [AuthConfig (0dfb279c544b51930363b74f9c094bd5282b8605d5b89971b817ad7ac67e95d7) AuthConfig (a474acfff2565f8b107c128bac895755b62fadc04d8c5361cfb26982839655e4) AuthConfig (9ca9d36d05295a7fe149c18cf778c8af3f3641ba59bfc952c2ae8ced324dd54b) AuthConfig (df67becca0f498b8b15358f6c309454d1c6e4309ffb989a85e75c2a5f997cb73) AuthConfig (1e4d690e0f7e5fdcec40cb088afd48595372c059365c5ab6d40f7bf998868f28) AuthConfig (55c88eaefe365547076fd1cfbfba73a8d08daf5333d2e7bc2a260d6406e7b0c8) AuthConfig (0e2761ca506777a19c32a3aa3a9ac71cd63ec7d9c4469a08dfcf53d45a726fb7) AuthConfig (3b2e3304ad0728c8197de416386777cc9d9520c00cebee3b46158713d607462a) AuthConfig (adfe8ec00d6b3d008c5b6935d19959fba7a2d27e5479f5697f8f044971635e95) AuthConfig (4e1c58168878c58f2246a1027d2f36eca0e8dfa22e022cb13ca2131317e7b205) AuthConfig (d0a5e1c4fd704f648d62c7bcfcc6d2a220bc153f841cc033af0f2a94fcc13e85) AuthConfig (609f5ca65ad4b48bba155787a66b474a2424a038c1efb7308c84d6aec86e5ee3) AuthConfig (e0e1ef48215cb0bb0ed010f3821057593448dc2784fbdd0e047dfdee508ca5ea) AuthConfig (d467fc629c3032d2e3876533570324782c2ad0beb8ba75f3dad6095923c15ea2) AuthConfig (b78c3a047d12038327f2fcdd35049bc882d96cf4704bc8dd73300811f830a1d9) AuthConfig (1d80346f8d05263593f54ddbe5d75f66b16002bc3e3bf62d4f6edddb737fc1b2) AuthConfig (b9da8a94e1df259b48c5017f1de85eccbfc81822d7b665efd86197f20b97e4b6) AuthConfig (8f2faa6bd4b4b5b190a10eace452e1efeb59269b176db8ad8013495275eb7c54)]', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-925e04-auth' namespace = 'ai-tenant-e2e-inf-a-925e04' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f59f8d67880> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-925e04-auth in ai-tenant-e2e-inf-a-925e04 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-925e04-auth","namespace":"ai-tenant-e2e-inf-a-925e04"},"spec":{"modelRefs":[{"name":"test-model-925e04","namespace":"ai-tenant-e2e-inf-a-925e04"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:06:27Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-925e04-auth', 'namespace': 'ai-tenant-e2e-inf-a-925e04', 'resourceVersion': '91617', 'uid': 'e24a9d2c-e90e-472c-8d3b-5a3a841b5f5b'}, 'spec': {'modelRefs': [{'name': 'test-model-925e04', 'namespace': 'ai-tenant-e2e-inf-a-925e04'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:06:27Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-a-925e04-maas-auth to be accepted and enforced: AuthPolicy waiting for the following components to sync: [AuthConfig (0dfb279c544b51930363b74f9c094bd5282b8605d5b89971b817ad7ac67e95d7) AuthConfig (a474acfff2565f8b107c128bac895755b62fadc04d8c5361cfb26982839655e4) AuthConfig (9ca9d36d05295a7fe149c18cf778c8af3f3641ba59bfc952c2ae8ced324dd54b) AuthConfig (df67becca0f498b8b15358f6c309454d1c6e4309ffb989a85e75c2a5f997cb73) AuthConfig (1e4d690e0f7e5fdcec40cb088afd48595372c059365c5ab6d40f7bf998868f28) AuthConfig (55c88eaefe365547076fd1cfbfba73a8d08daf5333d2e7bc2a260d6406e7b0c8) AuthConfig (0e2761ca506777a19c32a3aa3a9ac71cd63ec7d9c4469a08dfcf53d45a726fb7) AuthConfig (3b2e3304ad0728c8197de416386777cc9d9520c00cebee3b46158713d607462a) AuthConfig (adfe8ec00d6b3d008c5b6935d19959fba7a2d27e5479f5697f8f044971635e95) AuthConfig (4e1c58168878c58f2246a1027d2f36eca0e8dfa22e022cb13ca2131317e7b205) AuthConfig (d0a5e1c4fd704f648d62c7bcfcc6d2a220bc153f841cc033af0f2a94fcc13e85) AuthConfig (609f5ca65ad4b48bba155787a66b474a2424a038c1efb7308c84d6aec86e5ee3) AuthConfig (e0e1ef48215cb0bb0ed010f3821057593448dc2784fbdd0e047dfdee508ca5ea) AuthConfig (d467fc629c3032d2e3876533570324782c2ad0beb8ba75f3dad6095923c15ea2) AuthConfig (b78c3a047d12038327f2fcdd35049bc882d96cf4704bc8dd73300811f830a1d9) AuthConfig (1d80346f8d05263593f54ddbe5d75f66b16002bc3e3bf62d4f6edddb737fc1b2) AuthConfig (b9da8a94e1df259b48c5017f1de85eccbfc81822d7b665efd86197f20b97e4b6) AuthConfig (8f2faa6bd4b4b5b190a10eace452e1efeb59269b176db8ad8013495275eb7c54)]', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-7f7d21-auth' namespace = 'ai-tenant-e2e-inf-a-7f7d21' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f6f8562ad40> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-7f7d21-auth in ai-tenant-e2e-inf-a-7f7d21 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-7f7d21-auth","namespace":"ai-tenant-e2e-inf-a-7f7d21"},"spec":{"modelRefs":[{"name":"test-model-7f7d21","namespace":"ai-tenant-e2e-inf-a-7f7d21"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:06:48Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-7f7d21-auth', 'namespace': 'ai-tenant-e2e-inf-a-7f7d21', 'resourceVersion': '91586', 'uid': '1addd126-c25d-4367-9168-e25de55f9466'}, 'spec': {'modelRefs': [{'name': 'test-model-7f7d21', 'namespace': 'ai-tenant-e2e-inf-a-7f7d21'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:06:48Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-a-7f7d21-maas-auth to be accepted and enforced: AuthPolicy waiting for the following components to sync: [AuthConfig (5560df665312bb9dca306074a9b6227844d987345145723b54d759abf63076ac) AuthConfig (d49411d9bf5f5842196415e2231d86fc8b2ed6e3e91d469c62ed9cd28538aa9b) AuthConfig (5573be08a638bb13686a5e000e54ff8b08181e6a5f31843204b91166456edca6) AuthConfig (50574b42c088f2120778cb9a8028478aaca0818b3fc3ee25b5652590532bad2c) AuthConfig (1d6cc2639276311bf773fa92d584c3fe254a83d33c5f29a236356d0d594f0c3d) AuthConfig (dbec2270da43dd9998e5c1a0d16fd4f43335d74c6192f54d3b74240642cf820b) AuthConfig (9156bfea6d5cb3574bda4bcb1df19e5af58d53759d52b7eb7ddd6cd03ab19c61) AuthConfig (7f5925f8671ee0e4f4fe28d8a83ae64780d8aa2dc145bf8af682c3fc8a33c4a0) AuthConfig (bf0fa8872d88d4fd224bad9fd0151d906fc95fe8dba958fb0693a70264a74170) AuthConfig (e46abc8c21f85bb9340c8dd589a94bf391bf44905e2386629d089311a9ae3f8b) AuthConfig (66063a8e730da513cc1b7c8d0e6ccec1d3de791bb0184ae11ad2ca436686764b) AuthConfig (a4c3e0daef821e5947ef8836fa067075733ec293e36ae930270a0befc389edfd) AuthConfig (a0ce480bd36308443b4b2582e38e1ebdb36c444c72e40d778d9a3633ea09862f) AuthConfig (63029f53c63f912d0b7f18eff4dfae379e6870e0909f2b6b8d48da6e6d68868f) AuthConfig (1bd14a8195a63c95c4315a8ef125f5a82c1a4625650b7ff219aecf16fcbc9d54) AuthConfig (362a40b0df3e36b143719e2f3c1a36344d1ba5201b5a53acfa6c95fa9762bc03) AuthConfig (85705ce1a91340872cd15aa0dd1f8db4f2d5b470befae0ceb1b354cb7352c9d9) AuthConfig (aaf1cce9f0bc075225eeed107c40741cc7d83ffbe843d5d2762707c8891403f8)]', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-7f7d21-auth' namespace = 'ai-tenant-e2e-inf-a-7f7d21' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f6f8562ad40> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-7f7d21-auth in ai-tenant-e2e-inf-a-7f7d21 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-7f7d21-auth","namespace":"ai-tenant-e2e-inf-a-7f7d21"},"spec":{"modelRefs":[{"name":"test-model-7f7d21","namespace":"ai-tenant-e2e-inf-a-7f7d21"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:06:48Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-7f7d21-auth', 'namespace': 'ai-tenant-e2e-inf-a-7f7d21', 'resourceVersion': '91586', 'uid': '1addd126-c25d-4367-9168-e25de55f9466'}, 'spec': {'modelRefs': [{'name': 'test-model-7f7d21', 'namespace': 'ai-tenant-e2e-inf-a-7f7d21'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:06:48Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-a-7f7d21-maas-auth to be accepted and enforced: AuthPolicy waiting for the following components to sync: [AuthConfig (5560df665312bb9dca306074a9b6227844d987345145723b54d759abf63076ac) AuthConfig (d49411d9bf5f5842196415e2231d86fc8b2ed6e3e91d469c62ed9cd28538aa9b) AuthConfig (5573be08a638bb13686a5e000e54ff8b08181e6a5f31843204b91166456edca6) AuthConfig (50574b42c088f2120778cb9a8028478aaca0818b3fc3ee25b5652590532bad2c) AuthConfig (1d6cc2639276311bf773fa92d584c3fe254a83d33c5f29a236356d0d594f0c3d) AuthConfig (dbec2270da43dd9998e5c1a0d16fd4f43335d74c6192f54d3b74240642cf820b) AuthConfig (9156bfea6d5cb3574bda4bcb1df19e5af58d53759d52b7eb7ddd6cd03ab19c61) AuthConfig (7f5925f8671ee0e4f4fe28d8a83ae64780d8aa2dc145bf8af682c3fc8a33c4a0) AuthConfig (bf0fa8872d88d4fd224bad9fd0151d906fc95fe8dba958fb0693a70264a74170) AuthConfig (e46abc8c21f85bb9340c8dd589a94bf391bf44905e2386629d089311a9ae3f8b) AuthConfig (66063a8e730da513cc1b7c8d0e6ccec1d3de791bb0184ae11ad2ca436686764b) AuthConfig (a4c3e0daef821e5947ef8836fa067075733ec293e36ae930270a0befc389edfd) AuthConfig (a0ce480bd36308443b4b2582e38e1ebdb36c444c72e40d778d9a3633ea09862f) AuthConfig (63029f53c63f912d0b7f18eff4dfae379e6870e0909f2b6b8d48da6e6d68868f) AuthConfig (1bd14a8195a63c95c4315a8ef125f5a82c1a4625650b7ff219aecf16fcbc9d54) AuthConfig (362a40b0df3e36b143719e2f3c1a36344d1ba5201b5a53acfa6c95fa9762bc03) AuthConfig (85705ce1a91340872cd15aa0dd1f8db4f2d5b470befae0ceb1b354cb7352c9d9) AuthConfig (aaf1cce9f0bc075225eeed107c40741cc7d83ffbe843d5d2762707c8891403f8)]', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-db02a7-auth' namespace = 'ai-tenant-e2e-inf-b-db02a7' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7fe7057cf240> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-db02a7-auth in ai-tenant-e2e-inf-b-db02a7 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-db02a7-auth","namespace":"ai-tenant-e2e-inf-b-db02a7"},"spec":{"modelRefs":[{"name":"test-model-db02a7","namespace":"ai-tenant-e2e-inf-b-db02a7"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:06:59Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-db02a7-auth', 'namespace': 'ai-tenant-e2e-inf-b-db02a7', 'resourceVersion': '91663', 'uid': '0a833244-f381-4c8c-8121-b79fa6402a22'}, 'spec': {'modelRefs': [{'name': 'test-model-db02a7', 'namespace': 'ai-tenant-e2e-inf-b-db02a7'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:06:59Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-b-db02a7-maas-auth to be accepted and enforced: AuthPolicy waiting for the following components to sync: [AuthConfig (72950e72c64c8910840536a735a980ed64f8ffc97f20833af969ae285a640665) AuthConfig (c252550e9991ba38cba901d823a9648d6e1328b75193cdf968d3eebb00dc9e03) AuthConfig (b0fd3b4e38db78ab0805de2df0842bacbfbef7008eaa5a4b0c13de067acd0992) AuthConfig (96be850755aec5c28377824cc2aee161d8b5542b6df7d05b056d060fe73ad205) AuthConfig (4b4ba7135373a94083abac799b90c49de053f4271e5a9d27de64f9d9f3ab8d2c) AuthConfig (cf7f6de020ea0c5aeb184579516762493b4693169434d0bfc8d8487894933946) AuthConfig (abf659e404f8277b854f0628164063a21e2ee7d99ccbc95d0fcbb41301647ca1) AuthConfig (55def3d0f50b5352857a197a2228462c0ac16dd969d871f21a1af61e1aa41ff6) AuthConfig (27e307049d6906ac8f255319ce64f90bb80b5e67194e4e6d96ac9351215580db) AuthConfig (3daf6e7487de22538b63648b35c2279a6fa6c8e9c562f0ed64cea842266644d1) AuthConfig (de743f45a7a078631066b6b3e10d3fc26aa90c815fa3e851f94368e6a1316e44) AuthConfig (c3b634dd017b691a83dba6674ba84f10cce3baaae12c4cf9359737bedd61df45) AuthConfig (74f30ba10dcc2e2bee3cea68c84ed5bb683997262b90797072032763d94247c4) AuthConfig (669b26f5443db81b6c4fec74c45cb39af93a384283aa2904bb8ea9e6c8cc9d0f) AuthConfig (81f8a409a6def96973e97151394f4a2404cda8e91dc259a85ae77398d5690db4) AuthConfig (e06d022fa9dcd3c894f810c1201e69dcfae31812e4edbc8978b979e1ee4b5a93) AuthConfig (6e20192f0e4bc3d585f91a81ce20972848c8a2a21c6485b78d41064d0b45a845) AuthConfig (42382c5cd080e2555a672fa9d4b58643f18fcad1137a3fe63340f851965df626)]', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-db02a7-auth' namespace = 'ai-tenant-e2e-inf-b-db02a7' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7fe7057cf240> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-db02a7-auth in ai-tenant-e2e-inf-b-db02a7 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-db02a7-auth","namespace":"ai-tenant-e2e-inf-b-db02a7"},"spec":{"modelRefs":[{"name":"test-model-db02a7","namespace":"ai-tenant-e2e-inf-b-db02a7"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:06:59Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-db02a7-auth', 'namespace': 'ai-tenant-e2e-inf-b-db02a7', 'resourceVersion': '91663', 'uid': '0a833244-f381-4c8c-8121-b79fa6402a22'}, 'spec': {'modelRefs': [{'name': 'test-model-db02a7', 'namespace': 'ai-tenant-e2e-inf-b-db02a7'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:06:59Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-b-db02a7-maas-auth to be accepted and enforced: AuthPolicy waiting for the following components to sync: [AuthConfig (72950e72c64c8910840536a735a980ed64f8ffc97f20833af969ae285a640665) AuthConfig (c252550e9991ba38cba901d823a9648d6e1328b75193cdf968d3eebb00dc9e03) AuthConfig (b0fd3b4e38db78ab0805de2df0842bacbfbef7008eaa5a4b0c13de067acd0992) AuthConfig (96be850755aec5c28377824cc2aee161d8b5542b6df7d05b056d060fe73ad205) AuthConfig (4b4ba7135373a94083abac799b90c49de053f4271e5a9d27de64f9d9f3ab8d2c) AuthConfig (cf7f6de020ea0c5aeb184579516762493b4693169434d0bfc8d8487894933946) AuthConfig (abf659e404f8277b854f0628164063a21e2ee7d99ccbc95d0fcbb41301647ca1) AuthConfig (55def3d0f50b5352857a197a2228462c0ac16dd969d871f21a1af61e1aa41ff6) AuthConfig (27e307049d6906ac8f255319ce64f90bb80b5e67194e4e6d96ac9351215580db) AuthConfig (3daf6e7487de22538b63648b35c2279a6fa6c8e9c562f0ed64cea842266644d1) AuthConfig (de743f45a7a078631066b6b3e10d3fc26aa90c815fa3e851f94368e6a1316e44) AuthConfig (c3b634dd017b691a83dba6674ba84f10cce3baaae12c4cf9359737bedd61df45) AuthConfig (74f30ba10dcc2e2bee3cea68c84ed5bb683997262b90797072032763d94247c4) AuthConfig (669b26f5443db81b6c4fec74c45cb39af93a384283aa2904bb8ea9e6c8cc9d0f) AuthConfig (81f8a409a6def96973e97151394f4a2404cda8e91dc259a85ae77398d5690db4) AuthConfig (e06d022fa9dcd3c894f810c1201e69dcfae31812e4edbc8978b979e1ee4b5a93) AuthConfig (6e20192f0e4bc3d585f91a81ce20972848c8a2a21c6485b78d41064d0b45a845) AuthConfig (42382c5cd080e2555a672fa9d4b58643f18fcad1137a3fe63340f851965df626)]', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-1f5eee-auth' namespace = 'ai-tenant-e2e-inf-a-1f5eee' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7fe969aa5080> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-1f5eee-auth in ai-tenant-e2e-inf-a-1f5eee did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-1f5eee-auth","namespace":"ai-tenant-e2e-inf-a-1f5eee"},"spec":{"modelRefs":[{"name":"test-model-1f5eee","namespace":"ai-tenant-e2e-inf-a-1f5eee"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:06:34Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-1f5eee-auth', 'namespace': 'ai-tenant-e2e-inf-a-1f5eee', 'resourceVersion': '91570', 'uid': '0f6097d5-a16f-409f-9f4d-2032b2338e56'}, 'spec': {'modelRefs': [{'name': 'test-model-1f5eee', 'namespace': 'ai-tenant-e2e-inf-a-1f5eee'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:06:34Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-a-1f5eee-maas-auth to be accepted and enforced: AuthPolicy waiting for the following components to sync: [AuthConfig (1be0e34cf82afb7f753872497d2f07f9bce8373cd73ac20ab0fd3f81cc01a837) AuthConfig (8198cad31e5e7b47de2f23deaa0b5c3f683652a5a0fb5b7b6de8ffb9cab81cee) AuthConfig (008d86894516084673314bd5e2898ddfee6fb6363064e8b2f1114da33dbefb78) AuthConfig (a390049e1c861be426bcd02c9b250ae554e7c1e150c28d5a8b858404bad305ea) AuthConfig (b7c74f96da4631cf0491e34ad2c4dd639c6ffe6d9f5ea831f3b71e4f9576ebf5) AuthConfig (7fb7a6d5c545ad78eba5327d3e78f054bf68d4a3c511e1b3b94da72f633cfda5) AuthConfig (efd4d95a977f5bb2a9729ae7c80a9dc1944d3335cef86fb218b0614214910529) AuthConfig (481e488fdd982fca16e2272733b0d38d7f5f2d3e09ffd08e9986f45bdf56d1fc) AuthConfig (82d9bb0d8598f1502261fac19a2e3ef1c88ff645596061f5b8cf9f5ee6a86a7b) AuthConfig (36e12ea10bc7beb14775aea3fd63f3ff232c6994e39f6a7448cf9747f9173f66) AuthConfig (b51509ee8c9e9aedeb9d26d0e3fda3e23b0f413f42ce3cb9878fc1e9700ca72d) AuthConfig (55be06d20d6bf46200f05092bfb395bb354173b8055c519b618afec4a76dec8f) AuthConfig (6d62dc0ef931b3f70bc3f62e7ee3c664ed0f4459cf1037d86383299e6add8ca0) AuthConfig (6f0153f6e4f07df8485ca270d86b3677e5807b0b55ad0e8968992b4fbe0f3ea1) AuthConfig (5e853bfc8cb7f0e2c265674a991a96f058efa7a939477195ea05635d4d4af158) AuthConfig (ee28fcc8d0839ed880f58662cda22905700111f49280c647d426f97b84b7439c) AuthConfig (3373dada43bc86236dfc0d944c8e0859b7620bdeb1d278d8d1f14d56ee2019d9) AuthConfig (f215c42089bd20f13f90126404f20863f387033bbf69debf2ade28a41dff380e)]', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-1f5eee-auth' namespace = 'ai-tenant-e2e-inf-a-1f5eee' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7fe969aa5080> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-1f5eee-auth in ai-tenant-e2e-inf-a-1f5eee did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-1f5eee-auth","namespace":"ai-tenant-e2e-inf-a-1f5eee"},"spec":{"modelRefs":[{"name":"test-model-1f5eee","namespace":"ai-tenant-e2e-inf-a-1f5eee"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:06:34Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-1f5eee-auth', 'namespace': 'ai-tenant-e2e-inf-a-1f5eee', 'resourceVersion': '91570', 'uid': '0f6097d5-a16f-409f-9f4d-2032b2338e56'}, 'spec': {'modelRefs': [{'name': 'test-model-1f5eee', 'namespace': 'ai-tenant-e2e-inf-a-1f5eee'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:06:34Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-a-1f5eee-maas-auth to be accepted and enforced: AuthPolicy waiting for the following components to sync: [AuthConfig (1be0e34cf82afb7f753872497d2f07f9bce8373cd73ac20ab0fd3f81cc01a837) AuthConfig (8198cad31e5e7b47de2f23deaa0b5c3f683652a5a0fb5b7b6de8ffb9cab81cee) AuthConfig (008d86894516084673314bd5e2898ddfee6fb6363064e8b2f1114da33dbefb78) AuthConfig (a390049e1c861be426bcd02c9b250ae554e7c1e150c28d5a8b858404bad305ea) AuthConfig (b7c74f96da4631cf0491e34ad2c4dd639c6ffe6d9f5ea831f3b71e4f9576ebf5) AuthConfig (7fb7a6d5c545ad78eba5327d3e78f054bf68d4a3c511e1b3b94da72f633cfda5) AuthConfig (efd4d95a977f5bb2a9729ae7c80a9dc1944d3335cef86fb218b0614214910529) AuthConfig (481e488fdd982fca16e2272733b0d38d7f5f2d3e09ffd08e9986f45bdf56d1fc) AuthConfig (82d9bb0d8598f1502261fac19a2e3ef1c88ff645596061f5b8cf9f5ee6a86a7b) AuthConfig (36e12ea10bc7beb14775aea3fd63f3ff232c6994e39f6a7448cf9747f9173f66) AuthConfig (b51509ee8c9e9aedeb9d26d0e3fda3e23b0f413f42ce3cb9878fc1e9700ca72d) AuthConfig (55be06d20d6bf46200f05092bfb395bb354173b8055c519b618afec4a76dec8f) AuthConfig (6d62dc0ef931b3f70bc3f62e7ee3c664ed0f4459cf1037d86383299e6add8ca0) AuthConfig (6f0153f6e4f07df8485ca270d86b3677e5807b0b55ad0e8968992b4fbe0f3ea1) AuthConfig (5e853bfc8cb7f0e2c265674a991a96f058efa7a939477195ea05635d4d4af158) AuthConfig (ee28fcc8d0839ed880f58662cda22905700111f49280c647d426f97b84b7439c) AuthConfig (3373dada43bc86236dfc0d944c8e0859b7620bdeb1d278d8d1f14d56ee2019d9) AuthConfig (f215c42089bd20f13f90126404f20863f387033bbf69debf2ade28a41dff380e)]', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-d7a733-auth' namespace = 'ai-tenant-e2e-inf-b-d7a733' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f5fb27bb1a0> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-d7a733-auth in ai-tenant-e2e-inf-b-d7a733 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-d7a733-auth","namespace":"ai-tenant-e2e-inf-b-d7a733"},"spec":{"modelRefs":[{"name":"test-model-d7a733","namespace":"ai-tenant-e2e-inf-b-d7a733"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:07:58Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-d7a733-auth', 'namespace': 'ai-tenant-e2e-inf-b-d7a733', 'resourceVersion': '91606', 'uid': '7092e43b-42cd-4e4c-8a1b-c7821520944e'}, 'spec': {'modelRefs': [{'name': 'test-model-d7a733', 'namespace': 'ai-tenant-e2e-inf-b-d7a733'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:07:58Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-b-d7a733-maas-auth to be accepted and enforced: AuthPolicy status observed generation 1, current generation is 2', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-d7a733-auth' namespace = 'ai-tenant-e2e-inf-b-d7a733' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f5fb27bb1a0> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-d7a733-auth in ai-tenant-e2e-inf-b-d7a733 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-d7a733-auth","namespace":"ai-tenant-e2e-inf-b-d7a733"},"spec":{"modelRefs":[{"name":"test-model-d7a733","namespace":"ai-tenant-e2e-inf-b-d7a733"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:07:58Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-d7a733-auth', 'namespace': 'ai-tenant-e2e-inf-b-d7a733', 'resourceVersion': '91606', 'uid': '7092e43b-42cd-4e4c-8a1b-c7821520944e'}, 'spec': {'modelRefs': [{'name': 'test-model-d7a733', 'namespace': 'ai-tenant-e2e-inf-b-d7a733'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:07:58Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-b-d7a733-maas-auth to be accepted and enforced: AuthPolicy status observed generation 1, current generation is 2', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-82885f-auth' namespace = 'ai-tenant-e2e-inf-b-82885f' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f7abc5a6d40> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-82885f-auth in ai-tenant-e2e-inf-b-82885f did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-82885f-auth","namespace":"ai-tenant-e2e-inf-b-82885f"},"spec":{"modelRefs":[{"name":"test-model-82885f","namespace":"ai-tenant-e2e-inf-b-82885f"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:07:58Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-82885f-auth', 'namespace': 'ai-tenant-e2e-inf-b-82885f', 'resourceVersion': '91585', 'uid': 'ef94a6b7-1471-497b-900e-16733b3139fc'}, 'spec': {'modelRefs': [{'name': 'test-model-82885f', 'namespace': 'ai-tenant-e2e-inf-b-82885f'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:07:58Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-b-82885f-maas-auth to be accepted and enforced: AuthPolicy status observed generation 1, current generation is 2', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-82885f-auth' namespace = 'ai-tenant-e2e-inf-b-82885f' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f7abc5a6d40> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-82885f-auth in ai-tenant-e2e-inf-b-82885f did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-82885f-auth","namespace":"ai-tenant-e2e-inf-b-82885f"},"spec":{"modelRefs":[{"name":"test-model-82885f","namespace":"ai-tenant-e2e-inf-b-82885f"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:07:58Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-82885f-auth', 'namespace': 'ai-tenant-e2e-inf-b-82885f', 'resourceVersion': '91585', 'uid': 'ef94a6b7-1471-497b-900e-16733b3139fc'}, 'spec': {'modelRefs': [{'name': 'test-model-82885f', 'namespace': 'ai-tenant-e2e-inf-b-82885f'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:07:58Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-b-82885f-maas-auth to be accepted and enforced: AuthPolicy status observed generation 1, current generation is 2', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-77b4df-auth' namespace = 'ai-tenant-e2e-inf-b-77b4df' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f12b058c040> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-77b4df-auth in ai-tenant-e2e-inf-b-77b4df did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-77b4df-auth","namespace":"ai-tenant-e2e-inf-b-77b4df"},"spec":{"modelRefs":[{"name":"test-model-77b4df","namespace":"ai-tenant-e2e-inf-b-77b4df"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:07:59Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-77b4df-auth', 'namespace': 'ai-tenant-e2e-inf-b-77b4df', 'resourceVersion': '91649', 'uid': 'da2a63f4-b10f-490e-a3a0-960c99fa754b'}, 'spec': {'modelRefs': [{'name': 'test-model-77b4df', 'namespace': 'ai-tenant-e2e-inf-b-77b4df'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:07:59Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-b-77b4df-maas-auth to be accepted and enforced: AuthPolicy status observed generation 1, current generation is 2', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) > make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1296: in make_tenant_model_accessible wait_for_status_phase( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'test-model-77b4df-auth' namespace = 'ai-tenant-e2e-inf-b-77b4df' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f12b058c040> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/test-model-77b4df-auth in ai-tenant-e2e-inf-b-77b4df did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"test-model-77b4df-auth","namespace":"ai-tenant-e2e-inf-b-77b4df"},"spec":{"modelRefs":[{"name":"test-model-77b4df","namespace":"ai-tenant-e2e-inf-b-77b4df"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-10-30T20:07:59Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'test-model-77b4df-auth', 'namespace': 'ai-tenant-e2e-inf-b-77b4df', 'resourceVersion': '91649', 'uid': 'da2a63f4-b10f-490e-a3a0-960c99fa754b'}, 'spec': {'modelRefs': [{'name': 'test-model-77b4df', 'namespace': 'ai-tenant-e2e-inf-b-77b4df'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T20:07:59Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/e2e-inf-b-77b4df-maas-auth to be accepted and enforced: AuthPolicy status observed generation 1, current generation is 2', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError