/workspace/source/test/e2e/tests/test_api_keys.py:1429: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/e2e/tests/test_gateway_scoped_authpolicy.py:213: legacy gateway-default-auth is scoped to the shared default gateway/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_networkpolicy.py:119: AIGC praxis NetworkPolicy does not include OTLP 4317 egress (MaaS Go IPP bundle only)/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:231: praxis-extproc; Go IPP GATEWAY_NAME/TENANT_NAMESPACE env not applicable/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:262: praxis-extproc; EnvoyFilter uses payload-*-extproc clusters (see test_per_tenant_praxis_isolation)/workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:223: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:199: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:227: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:251: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:268: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:307: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:341: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:396: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:434: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_conflicts.py:362: client-facing model names are not resolvable through the gateway yet: MaaSModelRef e2e-extmod-cfl-2-2be3ef/e2e-cfl-same-130dc9 status.resolvedModelAlias='e2e-cfl-same-130dc9' != 'e2e-cfl-same-130dc9-two'self = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f55a5733e10> gateway_url = 'https://maas.apps.b80dc5000f3ec96c150e.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f55a55fd620> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.b80dc5000f3ec96c150e.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f55a55da3c0>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f55a55da3c0> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-225ee9'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-225ee9'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f55a5733e10> gateway_url = 'https://maas.apps.b80dc5000f3ec96c150e.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f55a55fd620> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.b80dc5000f3ec96c150e.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f55a55da3c0>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f55a55da3c0> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-225ee9'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-225ee9'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionError/workspace/source/test/e2e/tests/test_tenant_external_model_shared.py:165: client-facing model names are not resolvable through the gateway yet: MaaSModelRef e2e-extmod-shr-user-ad8174/e2e-shr-auth-8a65bf-e2e-extmod-a-88acce status.resolvedModelAlias='e2e-shr-auth-8a65bf' != 'e2e-shr-auth-8a65bf-client'self = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7fc32ef5fd90> gateway_url = 'https://maas.apps.b80dc5000f3ec96c150e.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7fc32ee23b00> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.b80dc5000f3ec96c150e.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7fc32ecd6840>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7fc32ecd6840> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-edd367'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-edd367'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionError/workspace/source/test/e2e/tests/test_tenant_external_model_shared.py:165: client-facing model names are not resolvable through the gateway yet: MaaSModelRef e2e-extmod-shr-user-f08338/e2e-shr-quota-9e567e-e2e-extmod-a-ad1661 status.resolvedModelAlias='e2e-shr-quota-9e567e' != 'e2e-shr-quota-9e567e-client'self = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f42e6eb7fd0> gateway_url = 'https://maas.apps.b80dc5000f3ec96c150e.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f42e6d7e7a0> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.b80dc5000f3ec96c150e.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f42e6d56040>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f42e6d56040> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-b38506'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-b38506'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7fe3df8bc110> gateway_url = 'https://maas.apps.b80dc5000f3ec96c150e.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7fe3df771d00> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.b80dc5000f3ec96c150e.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], '...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7fe3df7ca140>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7fe3df7ca140> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-f9db0f'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-f9db0f'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:268: AssertionError/workspace/source/test/e2e/tests/test_tenant_external_model_shared.py:165: client-facing model names are not resolvable through the gateway yet: MaaSModelRef e2e-extmod-shr-user-acd75d/e2e-shr-shared-262f36-e2e-extmod-a-2d2923 status.resolvedModelAlias='e2e-shr-shared-262f36' != 'e2e-shr-shared-262f36-client'