/workspace/source/test/maas-e2e/test/e2e/tests/test_api_keys.py:1429: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_subscription.py:1580: gateway-only mode: per-model AuthPolicy is not createdself = <test_subscription.TestE2ESubscriptionFlow object at 0x7f0fe5969410> def test_e2e_multiple_subscriptions_separate_keys_gets_200(self): """ User with two subscriptions for the same model: mint one API key per subscription; each key succeeds on inference without x-maas-subscription. """ ns = _ns() auth_policy_name = "e2e-test-auth-multi-sub-valid" subscription_1 = "e2e-test-subscription-free" subscription_2 = "e2e-test-subscription-premium" sa_name = "e2e-sa-multi-sub-valid" try: # Create service account and get OC token for maas-api oc_token = _create_sa_token(sa_name, namespace=ns) sa_user = _sa_to_user(sa_name, namespace=ns) # Create test resources with 2 subscriptions for the same user _create_test_auth_policy(auth_policy_name, MODEL_REF, users=[sa_user]) _create_test_subscription(subscription_1, MODEL_REF, users=[sa_user], token_limit=100) _create_test_subscription(subscription_2, MODEL_REF, users=[sa_user], token_limit=1000) _wait_for_maas_auth_policy_phase(auth_policy_name, require_enforced=False) > _wait_for_subscription_inference_ready( subscription_1, MODEL_REF, model_namespace=MODEL_NAMESPACE, timeout=180, ) test/maas-e2e/test/e2e/tests/test_subscription.py:2089: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_helper.py:1781: in _wait_for_subscription_inference_ready _wait_for_token_rate_limit_policy( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_ref = 'simulator-w1-e11f03' model_namespace = 'e2e-models-e2e-worker-w1-e11f03', timeout = 180 subscription_name = 'e2e-test-subscription-free' subscription_namespace = 'ai-tenant-e2e-worker-w1-e11f03' def _wait_for_token_rate_limit_policy( model_ref, model_namespace=MODEL_NAMESPACE, timeout=60, *, subscription_name=None, subscription_namespace=None ): """Wait for the current TokenRateLimitPolicy generation to be enforced. If a subscription is supplied, wait until the policy includes it. This prevents accepting the old policy before the controller updates its spec. Args: model_ref: Name of the model (e.g., "e2e-distinct-simulated") model_namespace: Namespace where the TRLP should be created (default: MODEL_NAMESPACE) timeout: Maximum wait time in seconds (default: 60) subscription_name: Subscription that the policy must include, if supplied subscription_namespace: Subscription namespace (defaults to _ns()) Raises: TimeoutError: If TRLP isn't created and enforced within timeout """ trlp_name = f"maas-trlp-{model_ref}" expected_subscription = ( f"{subscription_namespace or _ns()}/{subscription_name}" if subscription_name else None ) last_snapshot = "not found" deadline = time.time() + timeout log.info(f"Waiting for TokenRateLimitPolicy {trlp_name} in {model_namespace} (timeout: {timeout}s)...") while time.time() < deadline: result = subprocess.run( ["oc", "get", "tokenratelimitpolicy", trlp_name, "-n", model_namespace, "-o", "json"], capture_output=True, text=True, timeout=30, ) if result.returncode == 0: try: trlp = json.loads(result.stdout) metadata = trlp.get("metadata", {}) status = trlp.get("status", {}) generation = metadata.get("generation") observed = status.get("observedGeneration") conditions = status.get("conditions", []) enforced = next((c for c in conditions if c.get("type") == "Enforced"), None) subscriptions = { name.strip() for name in metadata.get("annotations", {}).get( "maas.opendatahub.io/subscriptions", "" ).split(",") } includes_subscription = expected_subscription is None or expected_subscription in subscriptions last_snapshot = ( f"generation={generation}, observedGeneration={observed}, " f"Enforced={enforced}, includes_subscription={includes_subscription}" ) if ( enforced and enforced.get("status") == "True" and generation is not None and observed == generation and includes_subscription ): log.info(f"TokenRateLimitPolicy {trlp_name} generation {generation} is enforced") return log.debug(f"TokenRateLimitPolicy {trlp_name} is not ready: {last_snapshot}") except (json.JSONDecodeError, KeyError) as e: log.debug(f"Failed to parse TRLP status: {e}") elif _is_not_found_error(result.stderr): log.debug(f"TokenRateLimitPolicy {trlp_name} not found yet...") elif _is_transient_kubectl_error(result.stderr): log.debug( f"Transient error while reading TokenRateLimitPolicy {trlp_name}: {result.stderr.strip()}" ) else: raise RuntimeError( f"Failed to get TokenRateLimitPolicy {trlp_name} in namespace '{model_namespace}': " f"{result.stderr.strip()}" ) time.sleep(3) > raise TimeoutError( f"TokenRateLimitPolicy {trlp_name} was not created and enforced in {model_namespace} within {timeout}s: {last_snapshot}" ) E TimeoutError: TokenRateLimitPolicy maas-trlp-simulator-w1-e11f03 was not created and enforced in e2e-models-e2e-worker-w1-e11f03 within 180s: generation=5, observedGeneration=5, Enforced={'lastTransitionTime': '2026-10-30T02:57:41Z', 'message': 'TokenRateLimitPolicy waiting for the following components to sync: [Limitador]', 'reason': 'Unknown', 'status': 'False', 'type': 'Enforced'}, includes_subscription=True test/maas-e2e/test/e2e/tests/test_helper.py:1538: TimeoutErrorapi_keys_base_url = 'https://maas.apps.bced66192e9659007ae2.prod.konflux-ocp-ci.dev/maas-api/v1/api-keys' headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IlRvbU8wVjkyTS1Oc2ZFcnZqNzZmRDU5OU1VeGgyN3BQSElPX1VRTENpbHMifQ.e...bj2SJH6Fmyw7Y1lEUEKyPknAavkxOmU7FCenRzNiJbIc96Q61KjwBwdgkVCwNHtYPxglhinLWUt7whZyw', 'Content-Type': 'application/json'} @pytest.fixture(scope="module") def api_key(api_keys_base_url: str, headers: dict) -> str: """ Create an API key for model inference tests. Returns the plaintext API key (show-once pattern). Note: The key inherits the authenticated user's groups, which should include system:authenticated to satisfy AuthPolicy requirements for model access. """ from multitenancy_helpers import response_summary sim_sub = os.environ.get("E2E_SIMULATOR_SUBSCRIPTION", "simulator-subscription") key_name = f"e2e-test-inference-key-{_xdist_worker_suffix()}" print(f"[api_key] Creating API key for inference tests ({key_name})...") r = requests.post( api_keys_base_url, headers=headers, json={"name": key_name, "subscription": sim_sub}, timeout=30, verify=TLS_VERIFY, ) # Accept both 200 and 201 as success if r.status_code not in (200, 201): > raise RuntimeError(f"Failed to create API key: {response_summary(r)}") E RuntimeError: Failed to create API key: status=500 x-ext-auth-reason=None body=Internal Server Error. test/maas-e2e/test/e2e/tests/conftest.py:372: RuntimeErrorapi_keys_base_url = 'https://maas.apps.bced66192e9659007ae2.prod.konflux-ocp-ci.dev/maas-api/v1/api-keys' headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IlRvbU8wVjkyTS1Oc2ZFcnZqNzZmRDU5OU1VeGgyN3BQSElPX1VRTENpbHMifQ.e...bj2SJH6Fmyw7Y1lEUEKyPknAavkxOmU7FCenRzNiJbIc96Q61KjwBwdgkVCwNHtYPxglhinLWUt7whZyw', 'Content-Type': 'application/json'} @pytest.fixture(scope="module") def api_key(api_keys_base_url: str, headers: dict) -> str: """ Create an API key for model inference tests. Returns the plaintext API key (show-once pattern). Note: The key inherits the authenticated user's groups, which should include system:authenticated to satisfy AuthPolicy requirements for model access. """ from multitenancy_helpers import response_summary sim_sub = os.environ.get("E2E_SIMULATOR_SUBSCRIPTION", "simulator-subscription") key_name = f"e2e-test-inference-key-{_xdist_worker_suffix()}" print(f"[api_key] Creating API key for inference tests ({key_name})...") r = requests.post( api_keys_base_url, headers=headers, json={"name": key_name, "subscription": sim_sub}, timeout=30, verify=TLS_VERIFY, ) # Accept both 200 and 201 as success if r.status_code not in (200, 201): > raise RuntimeError(f"Failed to create API key: {response_summary(r)}") E RuntimeError: Failed to create API key: status=500 x-ext-auth-reason=None body=Internal Server Error. test/maas-e2e/test/e2e/tests/conftest.py:372: RuntimeErrorapi_keys_base_url = 'https://maas.apps.bced66192e9659007ae2.prod.konflux-ocp-ci.dev/maas-api/v1/api-keys' headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IlRvbU8wVjkyTS1Oc2ZFcnZqNzZmRDU5OU1VeGgyN3BQSElPX1VRTENpbHMifQ.e...bj2SJH6Fmyw7Y1lEUEKyPknAavkxOmU7FCenRzNiJbIc96Q61KjwBwdgkVCwNHtYPxglhinLWUt7whZyw', 'Content-Type': 'application/json'} @pytest.fixture(scope="module") def api_key(api_keys_base_url: str, headers: dict) -> str: """ Create an API key for model inference tests. Returns the plaintext API key (show-once pattern). Note: The key inherits the authenticated user's groups, which should include system:authenticated to satisfy AuthPolicy requirements for model access. """ from multitenancy_helpers import response_summary sim_sub = os.environ.get("E2E_SIMULATOR_SUBSCRIPTION", "simulator-subscription") key_name = f"e2e-test-inference-key-{_xdist_worker_suffix()}" print(f"[api_key] Creating API key for inference tests ({key_name})...") r = requests.post( api_keys_base_url, headers=headers, json={"name": key_name, "subscription": sim_sub}, timeout=30, verify=TLS_VERIFY, ) # Accept both 200 and 201 as success if r.status_code not in (200, 201): > raise RuntimeError(f"Failed to create API key: {response_summary(r)}") E RuntimeError: Failed to create API key: status=500 x-ext-auth-reason=None body=Internal Server Error. test/maas-e2e/test/e2e/tests/conftest.py:372: RuntimeError/workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py:190: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deploy/workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py:213: legacy gateway-default-auth is scoped to the shared default gateway/workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py:223: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B/workspace/source/test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py:231: praxis-extproc; Go IPP GATEWAY_NAME/TENANT_NAMESPACE env not applicable/workspace/source/test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py:262: praxis-extproc; EnvoyFilter uses payload-*-extproc clusters (see test_per_tenant_praxis_isolation)