/workspace/source/test/e2e/tests/test_crd_watch_resilience.py:224: requires a disposable cluster: deletes the cluster-scoped KServe CRD; set E2E_RUN_CRD_WATCH_RESILIENCE=true to opt in/workspace/source/test/e2e/tests/test_crd_watch_resilience.py:262: requires a disposable cluster: deletes the cluster-scoped KServe CRD; set E2E_RUN_CRD_WATCH_RESILIENCE=true to opt in/workspace/source/test/e2e/tests/test_crd_watch_resilience.py:312: requires a disposable cluster: deletes the cluster-scoped KServe CRD; set E2E_RUN_CRD_WATCH_RESILIENCE=true to opt inself = <test_subscription.TestSubscriptionEnforcement object at 0x7fdf7b05e390> @pytest.mark.serial def test_models_endpoint_exempt_from_rate_limiting(self): """ Test that /v1/models endpoint remains accessible when token quota is exhausted. This verifies that users can discover model capabilities even when they've used all their inference tokens. The /v1/models endpoint is a discovery/metadata endpoint that does not consume tokens and should remain accessible. Ref: https://issues.redhat.com/browse/RHOAIENG-46770 Test steps: 1. Create subscription with very low token limit (15 tokens) 2. Exhaust the limit with inference requests (5 requests × 3 tokens = 15) 3. Verify inference requests get 429 (rate limited) 4. Verify /v1/models endpoint still returns 200 (not rate limited) """ # Use unconfigured model to isolate this test model_ref = UNCONFIGURED_MODEL_REF model_path = UNCONFIGURED_MODEL_PATH # Create unique subscription and auth policy names auth_policy_name = "e2e-models-exempt-test-auth" subscription_name = "e2e-models-exempt-test-subscription" # Very low limit for fast, deterministic test # With 3 token limit and max_tokens=1, we're guaranteed to exhaust quota within 5 requests # (even if each request uses exactly 1 token: 5 requests > 3 token limit) token_limit = 3 window = "1m" try: # 1. Create auth policy allowing system:authenticated _create_test_auth_policy( name=auth_policy_name, model_refs=[model_ref], groups=["system:authenticated"] ) _wait_for_maas_auth_policy_phase(auth_policy_name, require_auth_policies=False) # 2. Create subscription with low token limit _create_test_subscription( name=subscription_name, model_refs=[model_ref], groups=["system:authenticated"], token_limit=token_limit, window=window ) _wait_for_subscription_inference_ready( subscription_name, model_ref, model_namespace=MODEL_NAMESPACE, ) # 3. Create API key for this subscription oc_token = _get_cluster_token() api_key = _create_api_key( oc_token, name=f"e2e-models-exempt-{uuid.uuid4().hex[:8]}", subscription=subscription_name, ) # 4. Exhaust the token limit # With 3 token limit and 5 requests, we're guaranteed to hit the limit # (each successful request consumes ≥1 token, so 5 requests > 3 token limit) max_requests = 5 success_count = 0 rate_limited = False log.info(f"Exhausting token quota: sending up to {max_requests} requests") # Keep sending past the planned count until the enforcement deadline # so a limiter that trails the policy status does not fail the test. deadline = time.time() + RATE_LIMIT_ENFORCEMENT_TIMEOUT request_num = 0 while request_num < max_requests or time.time() < deadline: request_num += 1 r = _inference(api_key, path=model_path) log.info(f"Request {request_num}: status {r.status_code}") if r.status_code == 200: success_count += 1 elif r.status_code == 429: log.info(f"Rate limit hit after {success_count} successful requests") rate_limited = True break else: # Unexpected status during exhaustion log.warning(f"Unexpected status during quota exhaustion: {r.status_code}") # Verify we hit rate limit (otherwise test setup is broken) assert rate_limited, \ f"Expected to hit rate limit within {max_requests} requests with {token_limit} token limit, " \ f"but got {success_count} successful requests without hitting limit" # 5. Verify inference is now blocked with 429 log.info("Verifying inference endpoint is blocked...") r_inference = _inference(api_key, path=model_path) > assert r_inference.status_code == 429, \ f"Expected 429 for inference after exhausting tokens, got {r_inference.status_code}. " \ f"Response: {r_inference.text[:500]}" E AssertionError: Expected 429 for inference after exhausting tokens, got 200. Response: {"id":"cmpl-94522487-689a-5f26-9dcf-2bb1c537d905","created":1793423954,"model":"test/e2e-unconfigured-model","usage":{"prompt_tokens":1,"completion_tokens":3,"total_tokens":4},"object":"text_completion","kv_transfer_params":null,"choices":[{"index":0,"finish_reason":"length","text":"Testing@, "}]} E assert 200 == 429 E + where 200 = <Response [200]>.status_code test/e2e/tests/test_subscription.py:835: AssertionError/workspace/source/test/e2e/tests/test_subscription.py:961: no Kuadrant WasmPlugin for gateway maas-default-gateway/workspace/source/test/e2e/tests/test_subscription.py:1748: gateway-only mode: per-model AuthPolicy is not created/workspace/source/test/e2e/tests/test_tenant.py:401: default payload-processing is praxis-extproc; MaasTenantConfig.spec.payloadProcessing.resources is MaaS IPP-only/workspace/source/test/e2e/tests/test_tenant_external_model_compat.py:377: legacy IPP coexistence E2E requires the MaaS IPP image pin containing the legacy-controller exclusion guard; set TENANT_EXTERNAL_MODEL_LEGACY_IPP_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_compat.py:386: legacy IPP coexistence E2E requires the MaaS IPP image pin containing the legacy-controller exclusion guard; set TENANT_EXTERNAL_MODEL_LEGACY_IPP_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_compat.py:401: legacy IPP coexistence E2E requires the MaaS IPP image pin containing the legacy-controller exclusion guard; set TENANT_EXTERNAL_MODEL_LEGACY_IPP_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_compat.py:411: legacy IPP coexistence E2E requires the MaaS IPP image pin containing the legacy-controller exclusion guard; set TENANT_EXTERNAL_MODEL_LEGACY_IPP_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_compat.py:425: legacy IPP coexistence E2E requires the MaaS IPP image pin containing the legacy-controller exclusion guard; set TENANT_EXTERNAL_MODEL_LEGACY_IPP_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:359: Dormant-mode test mutates controller flags; set ENABLE_TENANT_DISCOVERY_DORMANT_E2E=true