/workspace/source/test/e2e/tests/test_crd_watch_resilience.py:224: requires a disposable cluster: deletes the cluster-scoped KServe CRD; set E2E_RUN_CRD_WATCH_RESILIENCE=true to opt in/workspace/source/test/e2e/tests/test_crd_watch_resilience.py:262: requires a disposable cluster: deletes the cluster-scoped KServe CRD; set E2E_RUN_CRD_WATCH_RESILIENCE=true to opt in/workspace/source/test/e2e/tests/test_crd_watch_resilience.py:312: requires a disposable cluster: deletes the cluster-scoped KServe CRD; set E2E_RUN_CRD_WATCH_RESILIENCE=true to opt inself = <test_subscription.TestSubscriptionEnforcement object at 0x7f02ac85e450> @pytest.mark.serial def test_rate_limit_exhaustion_gets_429(self): """ Test that a user gets 429 when they actually exceed their token rate limit. This test creates a dedicated subscription with a very low token limit, sends enough requests to exhaust it, and verifies a 429 response. Uses the unconfigured model to avoid interfering with other tests. """ # Use unconfigured model to isolate this test model_ref = UNCONFIGURED_MODEL_REF model_path = UNCONFIGURED_MODEL_PATH # Create unique subscription and auth policy names auth_policy_name = "e2e-rate-limit-test-auth" subscription_name = "e2e-rate-limit-test-subscription" # Low limit so we exhaust it quickly. Actual tokens consumed per # response are non-deterministic (max_tokens is a ceiling, not exact), # so we send enough requests to be confident we hit the limit without # asserting exactly when the 429 arrives. token_limit = 10 window = "1m" total_requests = 15 try: # 1. Create auth policy allowing system:authenticated _create_test_auth_policy( name=auth_policy_name, model_refs=[model_ref], groups=["system:authenticated"] ) _wait_for_maas_auth_policy_phase(auth_policy_name, require_enforced=False) # 2. Create subscription with low token limit _create_test_subscription( name=subscription_name, model_refs=[model_ref], groups=["system:authenticated"], token_limit=token_limit, window=window ) _wait_for_subscription_inference_ready( subscription_name, model_ref, model_namespace=MODEL_NAMESPACE, ) # 3. API key must be minted for this subscription oc_token = _get_cluster_token() api_key = _create_api_key( oc_token, name=f"e2e-rate-limit-{uuid.uuid4().hex[:8]}", subscription=subscription_name, ) # 4. Send requests to exhaust the limit. Past the planned count keep # going until the enforcement deadline: the limiter can trail the # policy status, and this test asserts a 429, not a status field. rate_limited = False success_count = 0 deadline = time.time() + RATE_LIMIT_ENFORCEMENT_TIMEOUT request_num = 0 while request_num < total_requests or time.time() < deadline: request_num += 1 r = _inference(api_key, path=model_path, max_tokens=1) log.info(f"Request {request_num}/{total_requests}: {r.status_code}") if r.status_code == 200: success_count += 1 elif r.status_code == 429: rate_limited = True log.info(f"Rate limit exceeded after {success_count} successful requests") # Verify it's a rate limit 429, not a subscription error response_text = r.text.lower() if r.text else "" # Rate limit 429s typically mention "rate", "limit", or "quota" # Subscription 429s mention "subscription" without "rate" is_rate_limit_error = any(keyword in response_text for keyword in ["rate", "limit", "quota", "too many"]) is_subscription_error = "subscription" in response_text and not is_rate_limit_error assert is_rate_limit_error or not is_subscription_error, \ f"Expected rate limit 429, not subscription error. Response: {r.text[:500]}" # Check for Retry-After header (optional but good practice) retry_after = r.headers.get("Retry-After") or r.headers.get("retry-after") if retry_after: log.info(f"Retry-After header present: {retry_after}") break else: # Unexpected status code raise AssertionError(f"Unexpected status {r.status_code} at request {request_num}: {r.text[:200]}") # Brief pause to avoid overwhelming the system, but stay within the window time.sleep(0.1) # Verify we actually exhausted the limit (at least one successful request) > assert success_count > 0, \ f"Got 429 on request #{request_num} without any successful requests. " \ f"This indicates a configuration issue, not rate limit exhaustion. Response: {r.text[:500]}" E AssertionError: Got 429 on request #1 without any successful requests. This indicates a configuration issue, not rate limit exhaustion. Response: Too Many Requests E E assert 0 > 0 test/e2e/tests/test_subscription.py:720: AssertionError/workspace/source/test/e2e/tests/test_subscription.py:961: no Kuadrant WasmPlugin for gateway maas-default-gateway/workspace/source/test/e2e/tests/test_subscription.py:1748: gateway-only mode: per-model AuthPolicy is not created/workspace/source/test/e2e/tests/test_tenant.py:401: default payload-processing is praxis-extproc; MaasTenantConfig.spec.payloadProcessing.resources is MaaS IPP-only/workspace/source/test/e2e/tests/test_tenant_external_model_compat.py:377: legacy IPP coexistence E2E requires the MaaS IPP image pin containing the legacy-controller exclusion guard; set TENANT_EXTERNAL_MODEL_LEGACY_IPP_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_compat.py:386: legacy IPP coexistence E2E requires the MaaS IPP image pin containing the legacy-controller exclusion guard; set TENANT_EXTERNAL_MODEL_LEGACY_IPP_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_compat.py:401: legacy IPP coexistence E2E requires the MaaS IPP image pin containing the legacy-controller exclusion guard; set TENANT_EXTERNAL_MODEL_LEGACY_IPP_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_compat.py:411: legacy IPP coexistence E2E requires the MaaS IPP image pin containing the legacy-controller exclusion guard; set TENANT_EXTERNAL_MODEL_LEGACY_IPP_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_compat.py:425: legacy IPP coexistence E2E requires the MaaS IPP image pin containing the legacy-controller exclusion guard; set TENANT_EXTERNAL_MODEL_LEGACY_IPP_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:359: Dormant-mode test mutates controller flags; set ENABLE_TENANT_DISCOVERY_DORMANT_E2E=true