/workspace/source/test/e2e/tests/test_api_keys.py:1429: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/e2e/tests/test_external_model_routing.py:564: provider withdrawal E2E for #106 is gated; set TENANT_EXTERNAL_MODEL_LAST_PROVIDER_WITHDRAWAL_E2E=true to enable (with it set, the #106 contract is expected to hold and a violation fails red)/workspace/source/test/e2e/tests/test_external_model_routing.py:481: provider withdrawal E2E for #108 is gated; set TENANT_EXTERNAL_MODEL_PARTIAL_WITHDRAWAL_E2E=true to enable (with it set, the #108 contract is expected to hold and a violation fails red)/workspace/source/test/e2e/tests/test_gateway_scoped_authpolicy.py:212: legacy gateway-default-auth is scoped to the shared default gateway/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_networkpolicy.py:119: AIGC praxis NetworkPolicy does not include OTLP 4317 egress (MaaS Go IPP bundle only)/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:231: praxis-extproc; Go IPP GATEWAY_NAME/TENANT_NAMESPACE env not applicable/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:262: praxis-extproc; EnvoyFilter uses payload-*-extproc clusters (see test_per_tenant_praxis_isolation)/workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:223: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B/workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:199: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:307: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:341: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:396: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:434: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:227: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enableself = <test_tenant_external_model_conflicts.TestForeignBrokenModelIsolation object at 0x7fc17f8ce610> conflicts_env = {'tenant_a': {'suffix': 'c738e0', 'tenant_ns': 'ai-tenant-e2e-extmod-a-c738e0', 'tenant_label_name': 'e2e-extmod-a-c73...tmod-cfl-2-388b84', 'resources': <tenant_external_model_helpers.TenantExternalModelResources object at 0x7fc17f76d410>} broken_pair = {'healthy': {'name': 'e2e-cfl-ok-5c8ce5', 'namespace': 'e2e-extmod-cfl-1-266342', 'client_name': 'e2e-cfl-ok-5c8ce5', ..._key': 'sk-oai-1dlMhENNdWllTNEq4_SphQeLknHvgRizFPEBDmhEoFFLQ3C9pdVSEibJij73X'}, 'broken_name': 'e2e-cfl-broken-496514'} def test_tenant_models_serve_after_broken_model_deleted(self, conflicts_env, broken_pair): """Deleting the broken model cleans up its failed attachment without disturbing the tenant's own serving model.""" env = conflicts_env tenant_a = env["tenant_a"] broken_ns = env["user_ns_2"] _delete_cr(EXTERNAL_MODEL_KIND, broken_pair["broken_name"], broken_ns) # _delete_cr only logs on failure: confirm the object is gone before # the absence checks, so a failed deletion cannot pass vacuously # (the broken model may have held no route to begin with). wait_for_not_found(EXTERNAL_MODEL_KIND, broken_pair["broken_name"], broken_ns) wait_for_no_attachment_httproute( broken_ns, tenant_a["gateway_name"], model_name=broken_pair["broken_name"] ) healthy = broken_pair["healthy"] response = infer_external_chat( tenant_a["gateway_url"], healthy["namespace"], healthy["client_name"], broken_pair["access"]["api_key"], path_tag=healthy["path_tag"], ) > assert_served_by_provider(response, healthy["path_tag"], credential=healthy["credential"]) test/e2e/tests/test_tenant_external_model_conflicts.py:481: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ response = <Response [503]>, path_tag = 'e2e-cfl-ok-5c8ce5-7092a5' credential = 'sk-extmod-7092a5' def assert_served_by_provider(response: requests.Response, path_tag: str, *, credential: str | None = None) -> None: """Assert a request reached the expected provider (and credential). The echo endpoint reflects the outgoing request URL and headers in the response body: the path tag confirms the path the client requested is the path the provider received (the gateway rewrites only the ``/{namespace}/{modelName}`` prefix), and the credential marker identifies the injected Secret — i.e. which provider binding served. """ > assert response.status_code == 200, ( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ f"expected 200 from external model inference, got {response.status_code}: " f"{redact_sensitive(response.text)}" ) E AssertionError: expected 200 from external model inference, got 503: test/e2e/tests/external_model_helpers.py:495: AssertionError/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:251: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:268: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable