/workspace/source/test/e2e/tests/test_api_keys.py:1429: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/e2e/tests/test_gateway_scoped_authpolicy.py:213: legacy gateway-default-auth is scoped to the shared default gateway/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_networkpolicy.py:119: AIGC praxis NetworkPolicy does not include OTLP 4317 egress (MaaS Go IPP bundle only)/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:262: praxis-extproc; EnvoyFilter uses payload-*-extproc clusters (see test_per_tenant_praxis_isolation)/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:231: praxis-extproc; Go IPP GATEWAY_NAME/TENANT_NAMESPACE env not applicable/workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:223: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B/workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:227: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:268: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:307: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:341: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:396: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:434: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_conflicts.py:362: client-facing model names are not resolvable through the gateway yet: MaaSModelRef e2e-extmod-cfl-2-b6a31d/e2e-cfl-same-651975 status.resolvedModelAlias='e2e-cfl-same-651975' != 'e2e-cfl-same-651975-two'/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:199: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:251: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_shared.py:165: client-facing model names are not resolvable through the gateway yet: MaaSModelRef e2e-extmod-shr-user-fe70b0/e2e-shr-shared-2bde33-e2e-extmod-a-76a19e status.resolvedModelAlias='e2e-shr-shared-2bde33' != 'e2e-shr-shared-2bde33-client'self = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f2b1af1e250> gateway_url = 'https://maas.apps.029a82205210b9bcf245.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f2b1ab1c180> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.029a82205210b9bcf245.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f2b1ab495c0>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f2b1ab495c0> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-ac15b3'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-ac15b3'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f2b1af1e250> gateway_url = 'https://maas.apps.029a82205210b9bcf245.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f2b1ab1c180> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.029a82205210b9bcf245.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f2b1ab495c0>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f2b1ab495c0> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-ac15b3'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-ac15b3'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionError/workspace/source/test/e2e/tests/test_tenant_external_model_shared.py:165: client-facing model names are not resolvable through the gateway yet: MaaSModelRef e2e-extmod-shr-user-fe70b0/e2e-shr-auth-9450cc-e2e-extmod-a-76a19e status.resolvedModelAlias='e2e-shr-auth-9450cc' != 'e2e-shr-auth-9450cc-client'self = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7fc863cb3bd0> gateway_url = 'https://maas.apps.029a82205210b9bcf245.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7fc863b6d4e0> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.029a82205210b9bcf245.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7fc863be7f80>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7fc863be7f80> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-1e90bc'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-1e90bc'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f5ec3164050> gateway_url = 'https://maas.apps.029a82205210b9bcf245.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f5ec301c5e0> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.029a82205210b9bcf245.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f5ec304e400>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f5ec304e400> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-b2e917'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-b2e917'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f22e17bc2d0> gateway_url = 'https://maas.apps.029a82205210b9bcf245.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f22e1671b20> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.029a82205210b9bcf245.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f22e16eef00>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f22e16eef00> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-4671d0'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-4671d0'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionError/workspace/source/test/e2e/tests/test_tenant_external_model_shared.py:165: client-facing model names are not resolvable through the gateway yet: MaaSModelRef e2e-extmod-shr-user-72f9d5/e2e-shr-quota-bd031a-e2e-extmod-a-817fd7 status.resolvedModelAlias='e2e-shr-quota-bd031a' != 'e2e-shr-quota-bd031a-client'