/workspace/source/test/e2e/tests/test_api_keys.py:1429: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/e2e/tests/test_external_model_routing.py:481: provider withdrawal E2E for #108 is gated; set TENANT_EXTERNAL_MODEL_PARTIAL_WITHDRAWAL_E2E=true to enable (with it set, the #108 contract is expected to hold and a violation fails red)/workspace/source/test/e2e/tests/test_external_model_routing.py:564: provider withdrawal E2E for #106 is gated; set TENANT_EXTERNAL_MODEL_LAST_PROVIDER_WITHDRAWAL_E2E=true to enable (with it set, the #106 contract is expected to hold and a violation fails red)/workspace/source/test/e2e/tests/test_gateway_scoped_authpolicy.py:212: legacy gateway-default-auth is scoped to the shared default gateway/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_networkpolicy.py:119: AIGC praxis NetworkPolicy does not include OTLP 4317 egress (MaaS Go IPP bundle only)/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:231: praxis-extproc; Go IPP GATEWAY_NAME/TENANT_NAMESPACE env not applicable/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:262: praxis-extproc; EnvoyFilter uses payload-*-extproc clusters (see test_per_tenant_praxis_isolation)/workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:223: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:199: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:307: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:341: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:396: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:434: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:268: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:251: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:227: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enableself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7fd5723de390> gateway_url = 'https://maas.apps.6e0fe14d1336b9c809cc.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7fd572184c20> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-length': ['109'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.6e0fe14d1336b9c809cc.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'user-ag...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7fd572147cc0>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7fd572147cc0> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-90f3ff'], 'content-length': ['109'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-90f3ff'], 'content-length': ['109'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:266: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7fd5723de390> gateway_url = 'https://maas.apps.6e0fe14d1336b9c809cc.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7fd572184c20> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-length': ['109'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.6e0fe14d1336b9c809cc.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'user-ag...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7fd572147cc0>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7fd572147cc0> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-90f3ff'], 'content-length': ['109'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-90f3ff'], 'content-length': ['109'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:266: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7fe9e02eef10> gateway_url = 'https://maas.apps.6e0fe14d1336b9c809cc.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7fe9e01aec00> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-length': ['109'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.6e0fe14d1336b9c809cc.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'user-ag...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7fe9dbf60b80>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7fe9dbf60b80> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-3b15a9'], 'content-length': ['109'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-3b15a9'], 'content-length': ['109'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:266: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7fe9e02eef10> gateway_url = 'https://maas.apps.6e0fe14d1336b9c809cc.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7fe9e01aec00> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-length': ['109'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.6e0fe14d1336b9c809cc.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'user-ag...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7fe9dbf60b80>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7fe9dbf60b80> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-3b15a9'], 'content-length': ['109'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-3b15a9'], 'content-length': ['109'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:266: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f7bdd3de310> gateway_url = 'https://maas.apps.6e0fe14d1336b9c809cc.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f7bdd2b9f80> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-length': ['109'], 'content-type': ['application/json'], 'forwarded': ['for=34.228.250.194;host=maas.apps.6e0fe14d1336b9c809cc.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'user-ag...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f7bdd1765c0>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f7bdd1765c0> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-35b8ad'], 'content-length': ['109'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-35b8ad'], 'content-length': ['109'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:266: AssertionError