/workspace/source/test/e2e/tests/test_api_keys.py:930: CronJob maas-api-key-cleanup not found in opendatahub: Error from server (NotFound): cronjobs.batch "maas-api-key-cleanup" not found/workspace/source/test/e2e/tests/test_api_keys.py:977: NetworkPolicy maas-api-cleanup-restrict not found in opendatahub: Error from server (NotFound): networkpolicies.networking.k8s.io "maas-api-cleanup-restrict" not found/workspace/source/test/e2e/tests/test_api_keys.py:1096: Cannot find maas-api pod in opendatahub: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}self = <test_api_keys.TestAPIKeySubscriptionPhases object at 0x7f5881130c10> def test_create_key_for_pending_subscription(self): """API key creation succeeds for Pending subscription.""" ns = _ns() subscription_name = "e2e-apikey-pending-sub" auth_name = "e2e-apikey-pending-auth" sa_name = "e2e-apikey-pending-sa" try: oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) _create_test_auth_policy(auth_name, MODEL_REF, users=[sa_user]) _create_test_subscription(subscription_name, MODEL_REF, users=[sa_user]) _wait_reconcile(seconds=10) # Patch to Pending phase patch_data = { "status": { "phase": "Pending", "conditions": [{ "type": "Ready", "status": "False", "reason": "Pending", "message": "Reconciliation in progress", "lastTransitionTime": datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ") }], } } cmd = [ "kubectl", "patch", "maassubscription", subscription_name, "-n", ns, "--type=merge", "--subresource=status", "-p", json.dumps(patch_data) ] result = subprocess.run(cmd, capture_output=True, text=True) assert result.returncode == 0, f"Failed to patch: {result.stderr}" cr = _get_cr("maassubscription", subscription_name, namespace=ns) phase = cr.get("status", {}).get("phase") > assert phase == "Pending", f"Expected Pending, got {phase}" E AssertionError: Expected Pending, got Active E assert 'Active' == 'Pending' E E - Pending E + Active test/e2e/tests/test_api_keys.py:1337: AssertionError/workspace/source/test/e2e/tests/test_namespace_scoping.py:212: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:245: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:283: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:320: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:377: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:453: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_subscription.py:1056: gateway-only mode: per-model AuthPolicy is not createdself = <test_models_endpoint.TestModelsEndpoint object at 0x7f5881113c40> def test_different_modelrefs_same_model_id(self): """ Test 7: Different modelRefs serving same model ID return separate entries. Uses two DIFFERENT MaaSModelRefs (each listed ONCE) that both serve the SAME model ID: - MODEL_REF (facebook-opt-125m-simulated) → serves "facebook/opt-125m" - PREMIUM_MODEL_REF (premium-simulated-simulated-premium) → serves "facebook/opt-125m" The API deduplicates by (model ID, URL). Since these are different backend services with different URLs, they return as 2 separate entries even though they serve the same model ID. Each entry shows the same model ID but different URL and subscription. """ log.info("Test 7: Different modelRefs same ID should deduplicate (INTENDED behavior)") sa_name = "e2e-models-diff-refs-sa" sa_ns = "default" maas_ns = _ns() subscription_name = "e2e-diff-refs-subscription" auth_policy_name = "e2e-diff-refs-auth" api_key = None try: # Create SA sa_token = _create_sa_token(sa_name, namespace=sa_ns) sa_user = _sa_to_user(sa_name, namespace=sa_ns) # Create auth policy with both modelRefs log.info(f"Creating auth policy with {MODEL_REF} and {PREMIUM_MODEL_REF}") auth_policy_cr = { "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSAuthPolicy", "metadata": { "name": auth_policy_name, "namespace": maas_ns, }, "spec": { "modelRefs": [ {"name": MODEL_REF, "namespace": MODEL_NAMESPACE}, {"name": PREMIUM_MODEL_REF, "namespace": MODEL_NAMESPACE}, ], "subjects": { "users": [sa_user], "groups": [], }, }, } subprocess.run( ["oc", "apply", "-f", "-"], input=json.dumps(auth_policy_cr), text=True, check=True, ) # Create subscription with both modelRefs (each listed ONCE) log.info(f"Creating subscription with {MODEL_REF} and {PREMIUM_MODEL_REF}") subscription_cr = { "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSSubscription", "metadata": { "name": subscription_name, "namespace": maas_ns, }, "spec": { "owner": { "users": [sa_user], "groups": [], }, "modelRefs": [ { "name": MODEL_REF, "namespace": MODEL_NAMESPACE, "tokenRateLimits": [{"limit": 100, "window": "1m"}], }, { "name": PREMIUM_MODEL_REF, "namespace": MODEL_NAMESPACE, "tokenRateLimits": [{"limit": 200, "window": "1m"}], }, ], }, } subprocess.run( ["oc", "apply", "-f", "-"], input=json.dumps(subscription_cr), text=True, check=True, ) # Wait for subscription to reconcile before creating API key _wait_for_maas_subscription_phase(subscription_name, namespace=maas_ns) # Create API key bound to our test subscription api_key = _create_api_key(sa_token, name="e2e-diff-refs-test-key", subscription=subscription_name) _wait_reconcile() # Query /v1/models log.info(f"Querying /v1/models with subscription: {subscription_name}") r = _get_models_with_gateway_retry( headers={ "Authorization": f"Bearer {api_key}", "x-maas-subscription": subscription_name, }, ) assert r.status_code == 200, f"Expected 200, got {r.status_code}: {r.text}" data = r.json() models = data.get("data") or [] assert isinstance(models, list), "Models should be a list" # Get model IDs from response model_ids = [m["id"] for m in models] unique_ids = set(model_ids) log.info(f"#x1F4CA API Response: {len(models)} total model(s), {len(unique_ids)} unique ID(s)") log.info(f" Model IDs: {model_ids}") log.info(f" Unique IDs: {unique_ids}") log.info(" Subscription had: 2 different modelRefs both serving 'facebook/opt-125m'") # Both modelRefs serve the same model ID assert len(unique_ids) == 1, \ f"Expected only 1 unique model ID (both modelRefs serve {MODEL_NAME}), got {len(unique_ids)}: {unique_ids}" # Verify it's the expected model ID expected_id = MODEL_NAME assert expected_id in unique_ids, \ f"Expected to find '{expected_id}', but got {unique_ids}" # INTENDED BEHAVIOR: Should return 2 entries (deduplication by model ID + URL) # Different backend services (different URLs) return separate entries even with same model ID > assert len(models) == 2, \ f"Expected 2 entries (different URLs), got {len(models)}: {model_ids}" E AssertionError: Expected 2 entries (different URLs), got 1: ['facebook/opt-125m'] E assert 1 == 2 E + where 1 = len([{'created': 1783341843, 'id': 'facebook/opt-125m', 'kind': 'LLMInferenceService', 'modelDetails': {'description': 'A simulated OPT-125M model for free-tier testing', 'displayName': 'Facebook OPT 125M (Simulated)'}, ...}]) test/e2e/tests/test_models_endpoint.py:856: AssertionError/workspace/source/test/e2e/tests/test_tenant.py:127: Tenant not Active (e.g. Degraded); payload-processing not asserted/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:186: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deploy/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:358: Dormant-mode test mutates controller flags; set ENABLE_TENANT_DISCOVERY_DORMANT_E2E=true/workspace/source/test/e2e/tests/test_gateway_scoped_authpolicy.py:170: gateway-default-auth not present (maas-gateway-auth is active); scoping is validated by unit tests/workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:158: OIDC_TOKEN_TENANT_A and OIDC_TOKEN_TENANT_B are required for per-tenant OIDC validation/workspace/source/test/e2e/tests/test_external_oidc.py:238: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:249: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:263: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:276: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:289: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:321: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:365: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:381: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:391: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:414: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:422: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:427: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:440: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:487: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:523: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:588: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:619: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:645: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:689: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:713: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:766: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:814: EXTERNAL_OIDC is not true