/workspace/source/test/maas-e2e/test/e2e/tests/test_api_keys.py:1429: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/maas-e2e/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=truemaas_api_base_url = 'https://maas.apps.9321a750a8b0878027a2.prod.konflux-ocp-ci.dev/maas-api' headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndRMlFFTTFiSE1zZjFfS0UtU1dDVEtRTmhWaWRaNm5tcDZvbDVnR2l4V28ifQ.e...ISeI94iGD3HRi2C9M-DzNiZauqPf4BDpmKhDHOWKzbCQxdjZKpIEh85GAedJb4E4TeyiKBQN-6oqdAwTQ', 'Content-Type': 'application/json'} @pytest.fixture(scope="session") def model_catalog(maas_api_base_url: str, headers: dict): r = requests.get(f"{maas_api_base_url}/v1/models", headers=headers, timeout=45, verify=TLS_VERIFY) > r.raise_for_status() test/maas-e2e/test/e2e/tests/conftest.py:289: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <Response [500]> def raise_for_status(self) -> None: """Raises :class:`HTTPError`, if one occurred.""" http_error_msg = "" if isinstance(self.reason, bytes): # We attempt to decode utf-8 first because some servers # choose to localize their reason strings. If the string # isn't utf-8, we fall back to iso-8859-1 for all other # encodings. (See PR #3538) try: reason = self.reason.decode("utf-8") except UnicodeDecodeError: reason = self.reason.decode("iso-8859-1") else: reason = self.reason if 400 <= self.status_code < 500: http_error_msg = ( f"{self.status_code} Client Error: {reason} for url: {self.url}" ) elif 500 <= self.status_code < 600: http_error_msg = ( f"{self.status_code} Server Error: {reason} for url: {self.url}" ) if http_error_msg: > raise HTTPError(http_error_msg, response=self) E requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://maas.apps.9321a750a8b0878027a2.prod.konflux-ocp-ci.dev/maas-api/v1/models test/e2e/.venv/lib64/python3.11/site-packages/requests/models.py:1167: HTTPErrormaas_api_base_url = 'https://maas.apps.9321a750a8b0878027a2.prod.konflux-ocp-ci.dev/maas-api' headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndRMlFFTTFiSE1zZjFfS0UtU1dDVEtRTmhWaWRaNm5tcDZvbDVnR2l4V28ifQ.e...ISeI94iGD3HRi2C9M-DzNiZauqPf4BDpmKhDHOWKzbCQxdjZKpIEh85GAedJb4E4TeyiKBQN-6oqdAwTQ', 'Content-Type': 'application/json'} @pytest.fixture(scope="session") def model_catalog(maas_api_base_url: str, headers: dict): r = requests.get(f"{maas_api_base_url}/v1/models", headers=headers, timeout=45, verify=TLS_VERIFY) > r.raise_for_status() test/maas-e2e/test/e2e/tests/conftest.py:289: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <Response [500]> def raise_for_status(self) -> None: """Raises :class:`HTTPError`, if one occurred.""" http_error_msg = "" if isinstance(self.reason, bytes): # We attempt to decode utf-8 first because some servers # choose to localize their reason strings. If the string # isn't utf-8, we fall back to iso-8859-1 for all other # encodings. (See PR #3538) try: reason = self.reason.decode("utf-8") except UnicodeDecodeError: reason = self.reason.decode("iso-8859-1") else: reason = self.reason if 400 <= self.status_code < 500: http_error_msg = ( f"{self.status_code} Client Error: {reason} for url: {self.url}" ) elif 500 <= self.status_code < 600: http_error_msg = ( f"{self.status_code} Server Error: {reason} for url: {self.url}" ) if http_error_msg: > raise HTTPError(http_error_msg, response=self) E requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://maas.apps.9321a750a8b0878027a2.prod.konflux-ocp-ci.dev/maas-api/v1/models test/e2e/.venv/lib64/python3.11/site-packages/requests/models.py:1167: HTTPErrormaas_api_base_url = 'https://maas.apps.9321a750a8b0878027a2.prod.konflux-ocp-ci.dev/maas-api' headers = {'Authorization': 'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IndRMlFFTTFiSE1zZjFfS0UtU1dDVEtRTmhWaWRaNm5tcDZvbDVnR2l4V28ifQ.e...ISeI94iGD3HRi2C9M-DzNiZauqPf4BDpmKhDHOWKzbCQxdjZKpIEh85GAedJb4E4TeyiKBQN-6oqdAwTQ', 'Content-Type': 'application/json'} @pytest.fixture(scope="session") def model_catalog(maas_api_base_url: str, headers: dict): r = requests.get(f"{maas_api_base_url}/v1/models", headers=headers, timeout=45, verify=TLS_VERIFY) > r.raise_for_status() test/maas-e2e/test/e2e/tests/conftest.py:289: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <Response [500]> def raise_for_status(self) -> None: """Raises :class:`HTTPError`, if one occurred.""" http_error_msg = "" if isinstance(self.reason, bytes): # We attempt to decode utf-8 first because some servers # choose to localize their reason strings. If the string # isn't utf-8, we fall back to iso-8859-1 for all other # encodings. (See PR #3538) try: reason = self.reason.decode("utf-8") except UnicodeDecodeError: reason = self.reason.decode("iso-8859-1") else: reason = self.reason if 400 <= self.status_code < 500: http_error_msg = ( f"{self.status_code} Client Error: {reason} for url: {self.url}" ) elif 500 <= self.status_code < 600: http_error_msg = ( f"{self.status_code} Server Error: {reason} for url: {self.url}" ) if http_error_msg: > raise HTTPError(http_error_msg, response=self) E requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://maas.apps.9321a750a8b0878027a2.prod.konflux-ocp-ci.dev/maas-api/v1/models test/e2e/.venv/lib64/python3.11/site-packages/requests/models.py:1167: HTTPError/workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py:190: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deployself = <test_tenant_namespace_discovery.TestTenantNamespaceDiscovery object at 0x7f2cd4a30710> def test_labeled_tenant_namespace_is_discovered(self): """1.1: Controller discovers labeled tenant namespaces and reconciles CRs.""" case = new_discovery_case(use_default_gateway=True) try: apply_discovery_labels(case["tenant_ns"], case["tenant_label_name"]) apply_tenant_cr(case["tenant_ns"], DEFAULT_GATEWAY_NAME) apply_maas_auth_policy(case["policy_name"], case["tenant_ns"]) apply_maas_subscription(case["subscription_name"], case["tenant_ns"]) wait_for_finalizer("maasauthpolicy", case["policy_name"], case["tenant_ns"], FINALIZER_AUTHPOLICY) wait_for_finalizer("maassubscription", case["subscription_name"], case["tenant_ns"], FINALIZER_SUBSCRIPTION) > auth = wait_for_status_phase( "maasauthpolicy", case["policy_name"], case["tenant_ns"], expected_phase="Active", ) test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py:92: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'e2e-policy-9e61f52c' namespace = 'ai-tenant-e2e-mt-9e61f52c' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f2cd4a1cd60> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/e2e-policy-9e61f52c in ai-tenant-e2e-mt-9e61f52c did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"e2e-policy-9e61f52c","namespace":"ai-tenant-e2e-mt-9e61f52c"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-11-03T19:44:32Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'e2e-policy-9e61f52c', 'namespace': 'ai-tenant-e2e-mt-9e61f52c', 'resourceVersion': '60452', 'uid': 'f40f37b7-1097-4a5b-a299-3a2aa441dd54'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-11-03T19:44:33Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/maas-gateway-auth to be accepted and enforced: AuthPolicy status observed generation 3, current generation is 51', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionErrorself = <test_tenant_namespace_discovery.TestLegacyDefaultNamespaceStillWorks object at 0x7f40088d7d50> def test_models_as_a_service_namespace_reconciles(self): ns = legacy_default_namespace() suffix = uuid.uuid4().hex[:8] policy_name = f"e2e-default-ns-{suffix}" try: apply_maas_auth_policy(policy_name, ns) wait_for_finalizer("maasauthpolicy", policy_name, ns, FINALIZER_AUTHPOLICY) > wait_for_status_phase("maasauthpolicy", policy_name, ns, expected_phase="Active") test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py:394: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'e2e-default-ns-0458b0f0' namespace = 'models-as-a-service' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f400844f1a0> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/e2e-default-ns-0458b0f0 in models-as-a-service did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"e2e-default-ns-0458b0f0","namespace":"models-as-a-service"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-11-03T19:44:53Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'e2e-default-ns-0458b0f0', 'namespace': 'models-as-a-service', 'resourceVersion': '61579', 'uid': 'e468f237-12e3-47c8-b5d8-c0432e9e83a4'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-11-03T19:46:38Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/maas-gateway-auth to be accepted and enforced: AuthPolicy status observed generation 3, current generation is 56', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionErrorself = <test_tenant_namespace_discovery.TestTenantNamespaceDiscovery object at 0x7f5a01f8fe10> def test_dynamic_discovery_after_label_added(self): """1.2 variant: Pre-existing CRs reconcile after namespace gains discovery labels.""" case = new_discovery_case(use_default_gateway=True) try: ensure_namespace(case["tenant_ns"]) apply_tenant_cr(case["tenant_ns"], DEFAULT_GATEWAY_NAME) apply_maas_auth_policy(case["policy_name"], case["tenant_ns"]) time.sleep(10) before = get_json_or_none("maasauthpolicy", case["policy_name"], case["tenant_ns"]) assert before is not None assert FINALIZER_AUTHPOLICY not in ((before.get("metadata") or {}).get("finalizers") or []) apply_discovery_labels(case["tenant_ns"], case["tenant_label_name"]) wait_for_finalizer("maasauthpolicy", case["policy_name"], case["tenant_ns"], FINALIZER_AUTHPOLICY) > wait_for_status_phase( "maasauthpolicy", case["policy_name"], case["tenant_ns"], expected_phase="Active", ) test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'e2e-policy-44c65306' namespace = 'ai-tenant-e2e-mt-44c65306' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f5a01eea2a0> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/e2e-policy-44c65306 in ai-tenant-e2e-mt-44c65306 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"e2e-policy-44c65306","namespace":"ai-tenant-e2e-mt-44c65306"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-11-03T19:45:19Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'e2e-policy-44c65306', 'namespace': 'ai-tenant-e2e-mt-44c65306', 'resourceVersion': '63229', 'uid': 'fb2ea167-c32d-4d62-9fae-c567b33e2831'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-11-03T19:45:30Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/maas-gateway-auth to be accepted and enforced: AuthPolicy status observed generation 3, current generation is 63', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError/workspace/source/test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py:212: legacy gateway-default-auth is scoped to the shared default gateway/workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners.self = <test_gateway_scoped_authpolicy.TestGatewayAuthPolicyLifecycle object at 0x7f71bda61f90> worker_tenant_context = WorkerTenantContext(worker_id='w4', suffix='d8a4ea', tenant_name='e2e-worker-w4-d8a4ea', tenant_namespace='ai-tenant-e...ulator-access', subscription_name='simulator-subscription', premium_subscription_name='premium-simulator-subscription') def test_gateway_auth_rego_is_fixed_size(self, worker_tenant_context): """6.3: Gateway auth rego is fixed-size — no model-specific data embedded.""" suffix = uuid.uuid4().hex[:8] policy_name = f"e2e-gw-auth-{suffix}" unique_group = f"e2e-gw-group-{suffix}" try: context = worker_tenant_context ap_before = get_gateway_authpolicy(name=context.gateway_authpolicy_name) gen_before = (ap_before or {}).get("metadata", {}).get("generation") _create_test_auth_policy( policy_name, context.model_ref, groups=[unique_group], namespace=context.tenant_namespace, model_namespace=context.model_namespace, ) > _wait_for_maas_auth_policy_phase( policy_name, namespace=context.tenant_namespace, require_auth_policies=False, ) test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py:98: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_helper.py:1897: in _wait_for_maas_auth_policy_phase _wait_for_gateway_auth_enforced(timeout=remaining) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'maas-gateway-auth', namespace = 'openshift-ingress', timeout = 240 def _wait_for_gateway_auth_enforced( name: Optional[str] = None, namespace: Optional[str] = None, timeout: Optional[int] = None, ): """Wait until the gateway Kuadrant AuthPolicy is Accepted and Enforced. MaaSAuthPolicy phase Active only means the controller reconciled CRs. HTTP calls through the gateway still need Kuadrant's AuthPolicy (typically maas-gateway-auth) to report Enforced=True; otherwise Envoy often returns an empty 403. Default timeout is GATEWAY_ENFORCED_TIMEOUT (env E2E_GATEWAY_ENFORCED_TIMEOUT). If the AuthPolicy CR is absent for GATEWAY_ENFORCED_MISSING_GRACE seconds, fail early — that usually means a wrong name/namespace, not slow enforcement. Raises: TimeoutError: with Accepted/Enforced snapshot so failures are actionable """ name = name or GATEWAY_AUTH_POLICY_NAME namespace = namespace or GATEWAY_NAMESPACE timeout = GATEWAY_ENFORCED_TIMEOUT if timeout is None else timeout deadline = time.time() + timeout last_snapshot = "AuthPolicy not found" missing_since = None warned_missing = False log.info( "Waiting for gateway AuthPolicy %s/%s Accepted+Enforced (timeout: %ds)...", namespace, name, timeout, ) while time.time() < deadline: remaining = max(0, int(deadline - time.time())) if remaining <= 0: break cr = _get_cr( "authpolicy", name, namespace, oc_timeout=min(OC_TIMEOUT, remaining), oc_retry_budget=remaining, ) if cr is None: last_snapshot = "AuthPolicy not found" now = time.time() if missing_since is None: missing_since = now if not warned_missing: log.warning( "Gateway AuthPolicy %s/%s not found yet; will fail after %ds if it never appears " "(check E2E_GATEWAY_AUTH_POLICY_NAME / GATEWAY_NAMESPACE)", namespace, name, GATEWAY_ENFORCED_MISSING_GRACE, ) warned_missing = True elif now - missing_since >= GATEWAY_ENFORCED_MISSING_GRACE: raise TimeoutError( f"Gateway AuthPolicy {namespace}/{name} was not found within " f"{GATEWAY_ENFORCED_MISSING_GRACE}s (misconfigured name/namespace?). " f"Empty HTTP 403 from maas-api usually means Kuadrant has not finished " f"enforcing auth on the gateway after MaaSAuthPolicy changes." ) else: missing_since = None conds = _authpolicy_conditions(cr, "Accepted", "Enforced") accepted, a_reason, a_msg = conds["Accepted"] enforced, e_reason, e_msg = conds["Enforced"] last_snapshot = ( f"Accepted={accepted} reason={a_reason!r} message={_truncate_auth_message(a_msg)!r}; " f"Enforced={enforced} reason={e_reason!r} message={_truncate_auth_message(e_msg)!r}" ) if accepted == "True" and enforced == "True": log.info("Gateway AuthPolicy %s/%s is Accepted and Enforced", namespace, name) return cr log.debug("Gateway AuthPolicy %s/%s not ready: %s", namespace, name, last_snapshot) sleep_for = min(2, max(0, int(deadline - time.time()))) if sleep_for > 0: time.sleep(sleep_for) > raise TimeoutError( f"Gateway AuthPolicy {namespace}/{name} was not Accepted+Enforced within {timeout}s " f"(last status: {last_snapshot}). Empty HTTP 403 from maas-api usually means Kuadrant " f"has not finished enforcing auth on the gateway after MaaSAuthPolicy changes." ) E TimeoutError: Gateway AuthPolicy openshift-ingress/maas-gateway-auth was not Accepted+Enforced within 240s (last status: Accepted=True reason='Accepted' message='AuthPolicy has been accepted'; Enforced=False reason='Unknown' message='AuthPolicy waiting for the following components to sync: [AuthConfig (319e246831bdaf5e765e63d5511b516594fd2b64527850e2ddabd2a4a85532ea) AuthConfig (7c924aa2b37ea38067b4e69052d541ea2e7c86fbdaa79b19f9ea09e87adbe4ff) AuthConfig (75fd86e2a4870a… (102 AuthConfigs pending sync)'). Empty HTTP 403 from maas-api usually means Kuadrant has not finished enforcing auth on the gateway after MaaSAuthPolicy changes. test/maas-e2e/test/e2e/tests/test_helper.py:1492: TimeoutError/workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners.