/workspace/source/test/e2e/tests/test_api_keys.py:1429: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/e2e/tests/test_external_model_routing.py:564: provider withdrawal E2E for #106 is gated; set TENANT_EXTERNAL_MODEL_LAST_PROVIDER_WITHDRAWAL_E2E=true to enable (with it set, the #106 contract is expected to hold and a violation fails red)/workspace/source/test/e2e/tests/test_external_model_routing.py:481: provider withdrawal E2E for #108 is gated; set TENANT_EXTERNAL_MODEL_PARTIAL_WITHDRAWAL_E2E=true to enable (with it set, the #108 contract is expected to hold and a violation fails red)/workspace/source/test/e2e/tests/test_gateway_scoped_authpolicy.py:212: legacy gateway-default-auth is scoped to the shared default gatewaycls = <class 'test_models_endpoint.TestModelsEndpoint'> @classmethod def setup_class(cls): """Validate test environment prerequisites before running any tests.""" log.info("=" * 60) log.info("Validating /v1/models E2E Test Prerequisites") log.info("=" * 60) # Validate MODEL_REF exists and is Ready model = _get_cr("maasmodelref", MODEL_REF, MODEL_NAMESPACE) if not model: pytest.fail(f"PREREQUISITE MISSING: MaaSModelRef '{MODEL_REF}' not found in namespace '{MODEL_NAMESPACE}'. " f"Ensure prow setup has created the model.") phase = model.get("status", {}).get("phase") endpoint = model.get("status", {}).get("endpoint") if phase != "Ready" or not endpoint: > pytest.fail(f"PREREQUISITE INVALID: MaaSModelRef '{MODEL_REF}' not Ready " f"(phase={phase}, endpoint={endpoint or 'none'}). " f"Wait for reconciliation or check controller logs.") E Failed: PREREQUISITE INVALID: MaaSModelRef 'simulator-w4-2a3e1b' not Ready (phase=Unhealthy, endpoint=none). Wait for reconciliation or check controller logs. test/e2e/tests/test_models_endpoint.py:381: Failedcls = <class 'test_models_endpoint.TestModelsEndpoint'> @classmethod def setup_class(cls): """Validate test environment prerequisites before running any tests.""" log.info("=" * 60) log.info("Validating /v1/models E2E Test Prerequisites") log.info("=" * 60) # Validate MODEL_REF exists and is Ready model = _get_cr("maasmodelref", MODEL_REF, MODEL_NAMESPACE) if not model: pytest.fail(f"PREREQUISITE MISSING: MaaSModelRef '{MODEL_REF}' not found in namespace '{MODEL_NAMESPACE}'. " f"Ensure prow setup has created the model.") phase = model.get("status", {}).get("phase") endpoint = model.get("status", {}).get("endpoint") if phase != "Ready" or not endpoint: > pytest.fail(f"PREREQUISITE INVALID: MaaSModelRef '{MODEL_REF}' not Ready " f"(phase={phase}, endpoint={endpoint or 'none'}). " f"Wait for reconciliation or check controller logs.") E Failed: PREREQUISITE INVALID: MaaSModelRef 'simulator-w4-2a3e1b' not Ready (phase=Unhealthy, endpoint=none). Wait for reconciliation or check controller logs. test/e2e/tests/test_models_endpoint.py:381: Failed/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_networkpolicy.py:119: AIGC praxis NetworkPolicy does not include OTLP 4317 egress (MaaS Go IPP bundle only)/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:231: praxis-extproc; Go IPP GATEWAY_NAME/TENANT_NAMESPACE env not applicable/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:262: praxis-extproc; EnvoyFilter uses payload-*-extproc clusters (see test_per_tenant_praxis_isolation)/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:223: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B/workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners.self = <test_tenant_external_model_attachment.TestListenerNotAllowed object at 0x7f5e13da6e90> attachment_env = {'tenant_a': {'suffix': '517027', 'tenant_ns': 'ai-tenant-e2e-extmod-a-517027', 'tenant_label_name': 'e2e-extmod-a-517...xtmod-b-5f9b3c', ...}, 'user_ns': 'e2e-extmod-att-user-b510c5', 'unadmitted_ns': 'e2e-extmod-att-noaccess-efed34', ...} refused_model = {'name': 'e2e-att-noaccess-c9aaea', 'namespace': 'e2e-extmod-att-noaccess-efed34', 'client_name': 'e2e-att-noaccess-c9aaea', 'model_refs': {}, ...} def test_route_not_admitted_and_not_served(self, attachment_env, refused_model): env = attachment_env model = refused_model tenant_b = env["tenant_b"] routes = attachment_httproutes(model["namespace"], tenant_b["gateway_name"]) if routes: # AGC creates the attachment's HTTPRoute in the model namespace # first; the gateway must refuse to admit it. for route in routes: conditions = httproute_parent_conditions(route, tenant_b["gateway_name"]) accepted = next((c for c in conditions if c.get("type") == "Accepted"), None) assert accepted is not None and accepted.get("status") == "False", ( f"route {route['metadata']['name']} must not be admitted: {conditions}" ) response = infer_external_chat( tenant_b["gateway_url"], model["namespace"], model["client_name"], api_key=f"sk-extmod-unused-{uuid.uuid4().hex[:8]}", path_tag=model["path_tag"], retries=1, ) > _assert_not_served(response) test/e2e/tests/test_tenant_external_model_attachment.py:476: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ response = <Response [503]> def _assert_not_served(response) -> None: """Assert a response proves the model was not served. A successful serve is a 200 from the echo provider, so any other status is a non-serving verdict. 401/403/404 are the expected refusals (auth denial / no route); 502/503 with a gateway-generated (non-provider) body are envoy-side failures on a gateway that holds no route for the model — observed under load while filters reload, and equally proof that no serve happened.""" assert response.status_code != 200, ( f"expected the model to not be served, got {response.status_code}: " f"{redact_sensitive(response.text)}" ) if response.status_code in (502, 503): > assert not (response.text or "").strip() or "Internal Server Error" in response.text or "Service Unavailable" in response.text, ( f"5xx carries a provider-shaped body, expected a gateway-generated " f"refusal/failure: {redact_sensitive(response.text)}" ) E AssertionError: 5xx carries a provider-shaped body, expected a gateway-generated refusal/failure: <html> E <head> E <meta name="viewport" content="width=device-width, initial-scale=1"> E E <style type="text/css"> E body { E font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; E line-height: 1.66666667; E font-size: 16px; E color: #333; E bac...<truncated> E assert (not '<html>\r\n <head>\r\n <meta name="viewport" content="width=device-width, initial-scale=1">\r\n\r\n <style type="text/css">\r\n body {\r\n font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\r\n line-height: 1.66666667;\r\n font-size: 16px;\r\n color: #333;\r\n background-color: #fff;\r\n margin: 2em 1em;\r\n }\r\n h1 {\r\n font-size: 28px;\r\n font-weight: 400;\r\n }\r\n p {\r\n margin: 0 0 10px;\r\n }\r\n .alert.alert-info {\r\n background-color: #F0F0F0;\r\n margin-top: 30px;\r\n padding: 30px;\r\n }\r\n .alert p {\r\n padding-left: 35px;\r\n }\r\n ul {\r\n padding-left: 51px;\r\n position: relative;\r\n }\r\n li {\r\n font-size: 14px;\r\n margin-bottom: 1em;\r\n }\r\n p.info {\r\n position: relative;\r\n font-size: 20px;\r\n }\r\n p.info:before, p.info:after {\r\n content: "";\r\n left: 0;\r\n position: absolute;\r\n top: 0;\r\n }\r\n p.info:before {\r\n background: #0066CC;\r\n border-radius: 1... margin: 6em;\r\n }\r\n }\r\n </style>\r\n </head>\r\n <body>\r\n <div>\r\n <h1>Application is not available</h1>\r\n <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>\r\n\r\n <div class="alert alert-info">\r\n <p class="info">\r\n Possible reasons you are seeing this page:\r\n </p>\r\n <ul>\r\n <li>\r\n <strong>The host doesn\'t exist.</strong>\r\n Make sure the hostname was typed correctly and that a route matching this hostname exists.\r\n </li>\r\n <li>\r\n <strong>The host exists, but doesn\'t have a matching path.</strong>\r\n Check if the URL path was typed correctly and that the route was created using the desired path.\r\n </li>\r\n <li>\r\n <strong>Route and path matches, but all pods are down.</strong>\r\n Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.\r\n </li>\r\n </ul>\r\n </div>\r\n </div>\r\n </body>\r\n</html>' or 'Internal Server Error' in '<html>\r\n <head>\r\n <meta name="viewport" content="width=device-width, initial-scale=1">\r\n\r\n <style type="text/css">\r\n body {\r\n font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\r\n line-height: 1.66666667;\r\n font-size: 16px;\r\n color: #333;\r\n background-color: #fff;\r\n margin: 2em 1em;\r\n }\r\n h1 {\r\n font-size: 28px;\r\n font-weight: 400;\r\n }\r\n p {\r\n margin: 0 0 10px;\r\n }\r\n .alert.alert-info {\r\n background-color: #F0F0F0;\r\n margin-top: 30px;\r\n padding: 30px;\r\n }\r\n .alert p {\r\n padding-left: 35px;\r\n }\r\n ul {\r\n padding-left: 51px;\r\n position: relative;\r\n }\r\n li {\r\n font-size: 14px;\r\n margin-bottom: 1em;\r\n }\r\n p.info {\r\n position: relative;\r\n font-size: 20px;\r\n }\r\n p.info:before, p.info:after {\r\n content: "";\r\n left: 0;\r\n position: absolute;\r\n top: 0;\r\n }\r\n p.info:before {\r\n background: #0066CC;\r\n border-radius: 1... margin: 6em;\r\n }\r\n }\r\n </style>\r\n </head>\r\n <body>\r\n <div>\r\n <h1>Application is not available</h1>\r\n <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>\r\n\r\n <div class="alert alert-info">\r\n <p class="info">\r\n Possible reasons you are seeing this page:\r\n </p>\r\n <ul>\r\n <li>\r\n <strong>The host doesn\'t exist.</strong>\r\n Make sure the hostname was typed correctly and that a route matching this hostname exists.\r\n </li>\r\n <li>\r\n <strong>The host exists, but doesn\'t have a matching path.</strong>\r\n Check if the URL path was typed correctly and that the route was created using the desired path.\r\n </li>\r\n <li>\r\n <strong>Route and path matches, but all pods are down.</strong>\r\n Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.\r\n </li>\r\n </ul>\r\n </div>\r\n </div>\r\n </body>\r\n</html>\r\n' or 'Service Unavailable' in '<html>\r\n <head>\r\n <meta name="viewport" content="width=device-width, initial-scale=1">\r\n\r\n <style type="text/css">\r\n body {\r\n font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\r\n line-height: 1.66666667;\r\n font-size: 16px;\r\n color: #333;\r\n background-color: #fff;\r\n margin: 2em 1em;\r\n }\r\n h1 {\r\n font-size: 28px;\r\n font-weight: 400;\r\n }\r\n p {\r\n margin: 0 0 10px;\r\n }\r\n .alert.alert-info {\r\n background-color: #F0F0F0;\r\n margin-top: 30px;\r\n padding: 30px;\r\n }\r\n .alert p {\r\n padding-left: 35px;\r\n }\r\n ul {\r\n padding-left: 51px;\r\n position: relative;\r\n }\r\n li {\r\n font-size: 14px;\r\n margin-bottom: 1em;\r\n }\r\n p.info {\r\n position: relative;\r\n font-size: 20px;\r\n }\r\n p.info:before, p.info:after {\r\n content: "";\r\n left: 0;\r\n position: absolute;\r\n top: 0;\r\n }\r\n p.info:before {\r\n background: #0066CC;\r\n border-radius: 1... margin: 6em;\r\n }\r\n }\r\n </style>\r\n </head>\r\n <body>\r\n <div>\r\n <h1>Application is not available</h1>\r\n <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>\r\n\r\n <div class="alert alert-info">\r\n <p class="info">\r\n Possible reasons you are seeing this page:\r\n </p>\r\n <ul>\r\n <li>\r\n <strong>The host doesn\'t exist.</strong>\r\n Make sure the hostname was typed correctly and that a route matching this hostname exists.\r\n </li>\r\n <li>\r\n <strong>The host exists, but doesn\'t have a matching path.</strong>\r\n Check if the URL path was typed correctly and that the route was created using the desired path.\r\n </li>\r\n <li>\r\n <strong>Route and path matches, but all pods are down.</strong>\r\n Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.\r\n </li>\r\n </ul>\r\n </div>\r\n </div>\r\n </body>\r\n</html>\r\n') E + where '<html>\r\n <head>\r\n <meta name="viewport" content="width=device-width, initial-scale=1">\r\n\r\n <style type="text/css">\r\n body {\r\n font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\r\n line-height: 1.66666667;\r\n font-size: 16px;\r\n color: #333;\r\n background-color: #fff;\r\n margin: 2em 1em;\r\n }\r\n h1 {\r\n font-size: 28px;\r\n font-weight: 400;\r\n }\r\n p {\r\n margin: 0 0 10px;\r\n }\r\n .alert.alert-info {\r\n background-color: #F0F0F0;\r\n margin-top: 30px;\r\n padding: 30px;\r\n }\r\n .alert p {\r\n padding-left: 35px;\r\n }\r\n ul {\r\n padding-left: 51px;\r\n position: relative;\r\n }\r\n li {\r\n font-size: 14px;\r\n margin-bottom: 1em;\r\n }\r\n p.info {\r\n position: relative;\r\n font-size: 20px;\r\n }\r\n p.info:before, p.info:after {\r\n content: "";\r\n left: 0;\r\n position: absolute;\r\n top: 0;\r\n }\r\n p.info:before {\r\n background: #0066CC;\r\n border-radius: 1... margin: 6em;\r\n }\r\n }\r\n </style>\r\n </head>\r\n <body>\r\n <div>\r\n <h1>Application is not available</h1>\r\n <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>\r\n\r\n <div class="alert alert-info">\r\n <p class="info">\r\n Possible reasons you are seeing this page:\r\n </p>\r\n <ul>\r\n <li>\r\n <strong>The host doesn\'t exist.</strong>\r\n Make sure the hostname was typed correctly and that a route matching this hostname exists.\r\n </li>\r\n <li>\r\n <strong>The host exists, but doesn\'t have a matching path.</strong>\r\n Check if the URL path was typed correctly and that the route was created using the desired path.\r\n </li>\r\n <li>\r\n <strong>Route and path matches, but all pods are down.</strong>\r\n Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.\r\n </li>\r\n </ul>\r\n </div>\r\n </div>\r\n </body>\r\n</html>' = <built-in method strip of str object at 0x55e9b4cbf8e0>() E + where <built-in method strip of str object at 0x55e9b4cbf8e0> = ('<html>\r\n <head>\r\n <meta name="viewport" content="width=device-width, initial-scale=1">\r\n\r\n <style type="text/css">\r\n body {\r\n font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\r\n line-height: 1.66666667;\r\n font-size: 16px;\r\n color: #333;\r\n background-color: #fff;\r\n margin: 2em 1em;\r\n }\r\n h1 {\r\n font-size: 28px;\r\n font-weight: 400;\r\n }\r\n p {\r\n margin: 0 0 10px;\r\n }\r\n .alert.alert-info {\r\n background-color: #F0F0F0;\r\n margin-top: 30px;\r\n padding: 30px;\r\n }\r\n .alert p {\r\n padding-left: 35px;\r\n }\r\n ul {\r\n padding-left: 51px;\r\n position: relative;\r\n }\r\n li {\r\n font-size: 14px;\r\n margin-bottom: 1em;\r\n }\r\n p.info {\r\n position: relative;\r\n font-size: 20px;\r\n }\r\n p.info:before, p.info:after {\r\n content: "";\r\n left: 0;\r\n position: absolute;\r\n top: 0;\r\n }\r\n p.info:before {\r\n background: #0066CC;\r\n border-radius: 1... margin: 6em;\r\n }\r\n }\r\n </style>\r\n </head>\r\n <body>\r\n <div>\r\n <h1>Application is not available</h1>\r\n <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>\r\n\r\n <div class="alert alert-info">\r\n <p class="info">\r\n Possible reasons you are seeing this page:\r\n </p>\r\n <ul>\r\n <li>\r\n <strong>The host doesn\'t exist.</strong>\r\n Make sure the hostname was typed correctly and that a route matching this hostname exists.\r\n </li>\r\n <li>\r\n <strong>The host exists, but doesn\'t have a matching path.</strong>\r\n Check if the URL path was typed correctly and that the route was created using the desired path.\r\n </li>\r\n <li>\r\n <strong>Route and path matches, but all pods are down.</strong>\r\n Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.\r\n </li>\r\n </ul>\r\n </div>\r\n </div>\r\n </body>\r\n</html>\r\n').strip E + where '<html>\r\n <head>\r\n <meta name="viewport" content="width=device-width, initial-scale=1">\r\n\r\n <style type="text/css">\r\n body {\r\n font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\r\n line-height: 1.66666667;\r\n font-size: 16px;\r\n color: #333;\r\n background-color: #fff;\r\n margin: 2em 1em;\r\n }\r\n h1 {\r\n font-size: 28px;\r\n font-weight: 400;\r\n }\r\n p {\r\n margin: 0 0 10px;\r\n }\r\n .alert.alert-info {\r\n background-color: #F0F0F0;\r\n margin-top: 30px;\r\n padding: 30px;\r\n }\r\n .alert p {\r\n padding-left: 35px;\r\n }\r\n ul {\r\n padding-left: 51px;\r\n position: relative;\r\n }\r\n li {\r\n font-size: 14px;\r\n margin-bottom: 1em;\r\n }\r\n p.info {\r\n position: relative;\r\n font-size: 20px;\r\n }\r\n p.info:before, p.info:after {\r\n content: "";\r\n left: 0;\r\n position: absolute;\r\n top: 0;\r\n }\r\n p.info:before {\r\n background: #0066CC;\r\n border-radius: 1... margin: 6em;\r\n }\r\n }\r\n </style>\r\n </head>\r\n <body>\r\n <div>\r\n <h1>Application is not available</h1>\r\n <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>\r\n\r\n <div class="alert alert-info">\r\n <p class="info">\r\n Possible reasons you are seeing this page:\r\n </p>\r\n <ul>\r\n <li>\r\n <strong>The host doesn\'t exist.</strong>\r\n Make sure the hostname was typed correctly and that a route matching this hostname exists.\r\n </li>\r\n <li>\r\n <strong>The host exists, but doesn\'t have a matching path.</strong>\r\n Check if the URL path was typed correctly and that the route was created using the desired path.\r\n </li>\r\n <li>\r\n <strong>Route and path matches, but all pods are down.</strong>\r\n Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.\r\n </li>\r\n </ul>\r\n </div>\r\n </div>\r\n </body>\r\n</html>\r\n' = <Response [503]>.text E + and '<html>\r\n <head>\r\n <meta name="viewport" content="width=device-width, initial-scale=1">\r\n\r\n <style type="text/css">\r\n body {\r\n font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\r\n line-height: 1.66666667;\r\n font-size: 16px;\r\n color: #333;\r\n background-color: #fff;\r\n margin: 2em 1em;\r\n }\r\n h1 {\r\n font-size: 28px;\r\n font-weight: 400;\r\n }\r\n p {\r\n margin: 0 0 10px;\r\n }\r\n .alert.alert-info {\r\n background-color: #F0F0F0;\r\n margin-top: 30px;\r\n padding: 30px;\r\n }\r\n .alert p {\r\n padding-left: 35px;\r\n }\r\n ul {\r\n padding-left: 51px;\r\n position: relative;\r\n }\r\n li {\r\n font-size: 14px;\r\n margin-bottom: 1em;\r\n }\r\n p.info {\r\n position: relative;\r\n font-size: 20px;\r\n }\r\n p.info:before, p.info:after {\r\n content: "";\r\n left: 0;\r\n position: absolute;\r\n top: 0;\r\n }\r\n p.info:before {\r\n background: #0066CC;\r\n border-radius: 1... margin: 6em;\r\n }\r\n }\r\n </style>\r\n </head>\r\n <body>\r\n <div>\r\n <h1>Application is not available</h1>\r\n <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>\r\n\r\n <div class="alert alert-info">\r\n <p class="info">\r\n Possible reasons you are seeing this page:\r\n </p>\r\n <ul>\r\n <li>\r\n <strong>The host doesn\'t exist.</strong>\r\n Make sure the hostname was typed correctly and that a route matching this hostname exists.\r\n </li>\r\n <li>\r\n <strong>The host exists, but doesn\'t have a matching path.</strong>\r\n Check if the URL path was typed correctly and that the route was created using the desired path.\r\n </li>\r\n <li>\r\n <strong>Route and path matches, but all pods are down.</strong>\r\n Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.\r\n </li>\r\n </ul>\r\n </div>\r\n </div>\r\n </body>\r\n</html>\r\n' = <Response [503]>.text E + and '<html>\r\n <head>\r\n <meta name="viewport" content="width=device-width, initial-scale=1">\r\n\r\n <style type="text/css">\r\n body {\r\n font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\r\n line-height: 1.66666667;\r\n font-size: 16px;\r\n color: #333;\r\n background-color: #fff;\r\n margin: 2em 1em;\r\n }\r\n h1 {\r\n font-size: 28px;\r\n font-weight: 400;\r\n }\r\n p {\r\n margin: 0 0 10px;\r\n }\r\n .alert.alert-info {\r\n background-color: #F0F0F0;\r\n margin-top: 30px;\r\n padding: 30px;\r\n }\r\n .alert p {\r\n padding-left: 35px;\r\n }\r\n ul {\r\n padding-left: 51px;\r\n position: relative;\r\n }\r\n li {\r\n font-size: 14px;\r\n margin-bottom: 1em;\r\n }\r\n p.info {\r\n position: relative;\r\n font-size: 20px;\r\n }\r\n p.info:before, p.info:after {\r\n content: "";\r\n left: 0;\r\n position: absolute;\r\n top: 0;\r\n }\r\n p.info:before {\r\n background: #0066CC;\r\n border-radius: 1... margin: 6em;\r\n }\r\n }\r\n </style>\r\n </head>\r\n <body>\r\n <div>\r\n <h1>Application is not available</h1>\r\n <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>\r\n\r\n <div class="alert alert-info">\r\n <p class="info">\r\n Possible reasons you are seeing this page:\r\n </p>\r\n <ul>\r\n <li>\r\n <strong>The host doesn\'t exist.</strong>\r\n Make sure the hostname was typed correctly and that a route matching this hostname exists.\r\n </li>\r\n <li>\r\n <strong>The host exists, but doesn\'t have a matching path.</strong>\r\n Check if the URL path was typed correctly and that the route was created using the desired path.\r\n </li>\r\n <li>\r\n <strong>Route and path matches, but all pods are down.</strong>\r\n Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.\r\n </li>\r\n </ul>\r\n </div>\r\n </div>\r\n </body>\r\n</html>\r\n' = <Response [503]>.text test/e2e/tests/test_tenant_external_model_attachment.py:135: AssertionError/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:268: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:341: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:396: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:434: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:251: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:199: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:227: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:307: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enableself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f5e13de5850> gateway_url = 'https://maas.apps.6807c75dec47df0b6041.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f5e13c96200> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=52.71.36.72;host=maas.apps.6807c75dec47df0b6041.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], 'use...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f5e13b09140>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f5e13b09140> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-1ef737'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-1ef737'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:266: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f5e13de5850> gateway_url = 'https://maas.apps.6807c75dec47df0b6041.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f5e13c96200> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=52.71.36.72;host=maas.apps.6807c75dec47df0b6041.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], 'use...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f5e13b09140>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f5e13b09140> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-1ef737'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-1ef737'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:266: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7fca79171850> gateway_url = 'https://maas.apps.6807c75dec47df0b6041.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7fca78ea9ee0> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=52.71.36.72;host=maas.apps.6807c75dec47df0b6041.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], 'use...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7fca78eedac0>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7fca78eedac0> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-f8adef'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-f8adef'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:266: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7fca79171850> gateway_url = 'https://maas.apps.6807c75dec47df0b6041.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7fca78ea9ee0> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=52.71.36.72;host=maas.apps.6807c75dec47df0b6041.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], 'use...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7fca78eedac0>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7fca78eedac0> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-f8adef'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-f8adef'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:266: AssertionErrorself = <test_tenant_external_model_default_tenant.TestDefaultTenantServesModelWithoutGatewayRefs object at 0x7f22c0bdd850> gateway_url = 'https://maas.apps.6807c75dec47df0b6041.prod.konflux-ocp-ci.dev' @pytest.fixture(scope="class") def implicit_model(self, gateway_url): tracked: list[tuple[str, str, str]] = [] try: model = _provision_model(tracked, "e2e-dft") attachment = _wait_implicitly_attached(model) > api_key = _serve_in_default_tenant(tracked, model, gateway_url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_default_tenant.py:391: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_default_tenant.py:317: in _serve_in_default_tenant _wait_for_subscription_header(gateway_url, model, api_key) test/e2e/tests/test_tenant_external_model_default_tenant.py:356: in _wait_for_subscription_header wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function _wait_for_subscription_header.<locals>._check at 0x7f22c0a8fe20> timeout = 300 what = 'Gateway never applied x-grid-geo-fencing header injection', interval = 5 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: Gateway never applied x-grid-geo-fencing header injection within 300s; last: provider headers: {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['<redacted>'], 'content-type': ['application/json'], 'forwarded': ['for=52.71.36.72;host=maas.apps.6807c75dec47df0b6041.prod.konflux-ocp-ci.dev;proto=https'], 'host': ['httpbingo.org'], 'transfer-encoding': ['chunked'], 'use...<truncated> E assert ['region1, region2'] == 'region1, region2' E + where ['region1, region2'] = <built-in method get of dict object at 0x7f22c0945680>('x-grid-geo-fencing') E + where <built-in method get of dict object at 0x7f22c0945680> = {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-af16f3'], 'content-type': ['application/json'], ...}.get E + where {'accept': ['*/*'], 'accept-encoding': ['gzip, deflate'], 'authorization': ['Bearer sk-extmod-af16f3'], 'content-type': ['application/json'], ...} = echo_headers(<Response [200]>) test/e2e/tests/multitenancy_helpers.py:266: AssertionError