/workspace/source/test/e2e/tests/test_api_keys.py:1429: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/e2e/tests/test_gateway_scoped_authpolicy.py:213: legacy gateway-default-auth is scoped to the shared default gateway/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_networkpolicy.py:119: AIGC praxis NetworkPolicy does not include OTLP 4317 egress (MaaS Go IPP bundle only)/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:231: praxis-extproc; Go IPP GATEWAY_NAME/TENANT_NAMESPACE env not applicable/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:262: praxis-extproc; EnvoyFilter uses payload-*-extproc clusters (see test_per_tenant_praxis_isolation)/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:223: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B/workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:199: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:227: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:251: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:268: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:307: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:341: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:396: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:434: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enableself = <test_tenant_external_model_conflicts.TestModelNameConflict object at 0x7f42e7ca3410> conflicts_env = {'tenant_a': {'suffix': '88cbc7', 'tenant_ns': 'ai-tenant-e2e-extmod-a-88cbc7', 'tenant_label_name': 'e2e-extmod-a-88c...tmod-cfl-2-b858a4', 'resources': <tenant_external_model_helpers.TenantExternalModelResources object at 0x7f42e7bc5bd0>} def test_tenant_own_model_wins_over_older_user_namespace_model(self, conflicts_env): """The tenant's own model wins even when a user-namespace model with the same client-facing name is older; the loser reports ModelNameConflict and never receives traffic.""" env = conflicts_env tenant_a = env["tenant_a"] gateway = gateway_ref(tenant_a["gateway_name"]) # One shared CR name — and therefore client-facing name — across a # user namespace and the tenant namespace. name = _conflict_cr_name() # The older model lives in a user namespace... older = provision_external_model_stack( env["resources"], name=name, namespace=env["user_ns_1"], gateway_refs=[gateway], ) # Guarantee the challenger is strictly older: creationTimestamp has # one-second resolution, and on a tie the AGC falls back to # namespace/name ordering — the test would risk passing through the # tiebreaker instead of the tenant-own rule. time.sleep(2) # ...the younger model lives in the tenant's own namespace. tenant_own = provision_external_model_stack( env["resources"], name=name, namespace=tenant_a["tenant_ns"], gateway_refs=[gateway], ) access = _serve_in_tenant(env, tenant_a, tenant_own) wait_for_gateway_attachment_reason( older["name"], older["namespace"], tenant_a["gateway_name"], "ModelNameConflict" ) # The conflict contract's serving effect: the loser is skipped by # admission, so it must hold no route on the gateway (matched by # route name — the namespace hosts other tests' models). wait_for_no_attachment_httproute( older["namespace"], tenant_a["gateway_name"], model_name=older["name"] ) # The winner serves with its own credential. response = infer_external_chat( tenant_a["gateway_url"], tenant_own["namespace"], tenant_own["client_name"], access["api_key"], path_tag=tenant_own["path_tag"], ) > assert_served_by_provider( response, tenant_own["path_tag"], credential=tenant_own["credential"] ) test/e2e/tests/test_tenant_external_model_conflicts.py:163: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ response = <Response [503]>, path_tag = 'e2e-cfl-dfc25b-65bb39' credential = 'sk-extmod-65bb39' def assert_served_by_provider(response: requests.Response, path_tag: str, *, credential: str | None = None) -> None: """Assert a request reached the expected provider (and credential). The echo endpoint reflects the outgoing request URL and headers in the response body: the path tag confirms the path the client requested is the path the provider received (the gateway rewrites only the ``/{namespace}/{modelName}`` prefix), and the credential marker identifies the injected Secret — i.e. which provider binding served. """ > assert response.status_code == 200, ( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ f"expected 200 from external model inference, got {response.status_code}: " f"{redact_sensitive(response.text)}" ) E AssertionError: expected 200 from external model inference, got 503: test/e2e/tests/tenant_external_model_helpers.py:1536: AssertionError/workspace/source/test/e2e/tests/test_tenant_external_model_conflicts.py:362: client-facing model names are not resolvable through the gateway yet: MaaSModelRef e2e-extmod-cfl-2-d1e48c/e2e-cfl-same-afe760 status.resolvedModelAlias='e2e-cfl-same-afe760' != 'e2e-cfl-same-afe760-two'self = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f4617ad0850> gateway_url = 'https://maas.apps.b87e053679e6e3bf870a.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f4617824040> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.b87e053679e6e3bf870a.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f461782cb00>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f461782cb00> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-75aa5f'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-75aa5f'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f4617ad0850> gateway_url = 'https://maas.apps.b87e053679e6e3bf870a.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f4617824040> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.b87e053679e6e3bf870a.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f461782cb00>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f461782cb00> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-75aa5f'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-75aa5f'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f77c4210250> gateway_url = 'https://maas.apps.b87e053679e6e3bf870a.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f77bfe91d00> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.b87e053679e6e3bf870a.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f77bfd0d8c0>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f77bfd0d8c0> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-026515'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-026515'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f77c4210250> gateway_url = 'https://maas.apps.b87e053679e6e3bf870a.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f77bfe91d00> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.b87e053679e6e3bf870a.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f77bfd0d8c0>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f77bfd0d8c0> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-026515'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-026515'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionError/workspace/source/test/e2e/tests/test_tenant_external_model_shared.py:165: client-facing model names are not resolvable through the gateway yet: MaaSModelRef e2e-extmod-shr-user-d7552b/e2e-shr-shared-29b8bf-e2e-extmod-a-d60c68 status.resolvedModelAlias='e2e-shr-shared-29b8bf' != 'e2e-shr-shared-29b8bf-client'/workspace/source/test/e2e/tests/test_tenant_external_model_shared.py:165: client-facing model names are not resolvable through the gateway yet: MaaSModelRef e2e-extmod-shr-user-0ab35b/e2e-shr-auth-b9719c-e2e-extmod-a-88cbc7 status.resolvedModelAlias='e2e-shr-auth-b9719c' != 'e2e-shr-auth-b9719c-client'self = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7fe75d7c81d0> gateway_url = 'https://maas.apps.b87e053679e6e3bf870a.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7fe75d6828e0> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.b87e053679e6e3bf870a.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7fe75d548c00>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7fe75d548c00> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-eb465f'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-eb465f'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionError/workspace/source/test/e2e/tests/test_tenant_external_model_shared.py:165: client-facing model names are not resolvable through the gateway yet: MaaSModelRef e2e-extmod-shr-user-0ab35b/e2e-shr-quota-2dda93-e2e-extmod-a-88cbc7 status.resolvedModelAlias='e2e-shr-quota-2dda93' != 'e2e-shr-quota-2dda93-client'