/workspace/source/test/e2e/tests/test_api_keys.py:1429: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/e2e/tests/test_external_model_routing.py:481: provider withdrawal E2E for #108 is gated; set TENANT_EXTERNAL_MODEL_PARTIAL_WITHDRAWAL_E2E=true to enable (with it set, the #108 contract is expected to hold and a violation fails red)/workspace/source/test/e2e/tests/test_gateway_scoped_authpolicy.py:212: legacy gateway-default-auth is scoped to the shared default gateway/workspace/source/test/e2e/tests/test_external_model_routing.py:564: provider withdrawal E2E for #106 is gated; set TENANT_EXTERNAL_MODEL_LAST_PROVIDER_WITHDRAWAL_E2E=true to enable (with it set, the #106 contract is expected to hold and a violation fails red)/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_networkpolicy.py:119: AIGC praxis NetworkPolicy does not include OTLP 4317 egress (MaaS Go IPP bundle only)/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:231: praxis-extproc; Go IPP GATEWAY_NAME/TENANT_NAMESPACE env not applicable/workspace/source/test/e2e/tests/test_per_tenant_ipp_isolation.py:262: praxis-extproc; EnvoyFilter uses payload-*-extproc clusters (see test_per_tenant_praxis_isolation)/workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:223: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:199: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:251: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:268: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:307: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:341: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:396: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:434: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable/workspace/source/test/e2e/tests/test_tenant_external_model_crossns.py:227: cross-namespace provider E2E is gated until AGC ships cross-namespace provider support; set TENANT_EXTERNAL_MODEL_CROSSNS_E2E=true to enable@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1130: in bootstrap_aitenant_tenant apply_gateway_route_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) test/e2e/tests/multitenancy_helpers.py:1013: in apply_gateway_route_fixture wait_for_route_admitted(route_name) test/e2e/tests/multitenancy_helpers.py:812: in wait_for_route_admitted return wait_for_json("route", route_name, GATEWAY_NAMESPACE, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'route', name = 'e2e-inf-a-759b04-route', namespace = 'openshift-ingress' predicate = <function wait_for_route_admitted.<locals>._predicate at 0x7fd0b24d8f40> timeout = 60, interval = 3 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: route/e2e-inf-a-759b04-route in openshift-ingress did not satisfy condition. Last object: {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"gateway","app.kubernetes.io/instance":"e2e-inf-a-759b04","app.kubernetes.io/name":"maas","e2e.maas.opendatahub.io/fixture":"e2e-inf-a-759b04","gateway.networking.k8s.io/gateway-name":"e2e-inf-a-759b04"},"name":"e2e-inf-a-759b04-route","namespace":"openshift-ingress"},"spec":{"host":"e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev","port":{"targetPort":443},"tls":{"destinationCACertificate":"-----BEGIN CERTIFICATE-----\\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\\n-----END CERTIFICATE-----\\n","insecureEdgeTerminationPolicy":"Redirect","termination":"reencrypt"},"to":{"kind":"Service","name":"e2e-inf-a-759b04-openshift-default","weight":100}}}\n'}, 'creationTimestamp': '2026-11-03T20:26:27Z', 'generation': 1, 'labels': {'app.kubernetes.io/component': 'gateway', 'app.kubernetes.io/instance': 'e2e-inf-a-759b04', 'app.kubernetes.io/name': 'maas', 'e2e.maas.opendatahub.io/fixture': 'e2e-inf-a-759b04', 'gateway.networking.k8s.io/gateway-name': 'e2e-inf-a-759b04'}, 'name': 'e2e-inf-a-759b04-route', 'namespace': 'openshift-ingress', 'resourceVersion': '157101', 'uid': '9b60ffb8-33dc-44d8-95d2-f167cad41f26'}, 'spec': {'host': 'e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', 'port': {'targetPort': 443}, 'tls': {'destinationCACertificate': '-----BEGIN CERTIFICATE-----\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\n-----END CERTIFICATE-----\n', 'insecureEdgeTerminationPolicy': 'Redirect', 'termination': 'reencrypt'}, 'to': {'kind': 'Service', 'name': 'e2e-inf-a-759b04-openshift-default', 'weight': 100}, 'wildcardPolicy': 'None'}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:301: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1130: in bootstrap_aitenant_tenant apply_gateway_route_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) test/e2e/tests/multitenancy_helpers.py:1013: in apply_gateway_route_fixture wait_for_route_admitted(route_name) test/e2e/tests/multitenancy_helpers.py:812: in wait_for_route_admitted return wait_for_json("route", route_name, GATEWAY_NAMESPACE, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'route', name = 'e2e-inf-a-759b04-route', namespace = 'openshift-ingress' predicate = <function wait_for_route_admitted.<locals>._predicate at 0x7fd0b24d8f40> timeout = 60, interval = 3 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: route/e2e-inf-a-759b04-route in openshift-ingress did not satisfy condition. Last object: {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"gateway","app.kubernetes.io/instance":"e2e-inf-a-759b04","app.kubernetes.io/name":"maas","e2e.maas.opendatahub.io/fixture":"e2e-inf-a-759b04","gateway.networking.k8s.io/gateway-name":"e2e-inf-a-759b04"},"name":"e2e-inf-a-759b04-route","namespace":"openshift-ingress"},"spec":{"host":"e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev","port":{"targetPort":443},"tls":{"destinationCACertificate":"-----BEGIN CERTIFICATE-----\\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\\n-----END CERTIFICATE-----\\n","insecureEdgeTerminationPolicy":"Redirect","termination":"reencrypt"},"to":{"kind":"Service","name":"e2e-inf-a-759b04-openshift-default","weight":100}}}\n'}, 'creationTimestamp': '2026-11-03T20:26:27Z', 'generation': 1, 'labels': {'app.kubernetes.io/component': 'gateway', 'app.kubernetes.io/instance': 'e2e-inf-a-759b04', 'app.kubernetes.io/name': 'maas', 'e2e.maas.opendatahub.io/fixture': 'e2e-inf-a-759b04', 'gateway.networking.k8s.io/gateway-name': 'e2e-inf-a-759b04'}, 'name': 'e2e-inf-a-759b04-route', 'namespace': 'openshift-ingress', 'resourceVersion': '157101', 'uid': '9b60ffb8-33dc-44d8-95d2-f167cad41f26'}, 'spec': {'host': 'e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', 'port': {'targetPort': 443}, 'tls': {'destinationCACertificate': '-----BEGIN CERTIFICATE-----\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\n-----END CERTIFICATE-----\n', 'insecureEdgeTerminationPolicy': 'Redirect', 'termination': 'reencrypt'}, 'to': {'kind': 'Service', 'name': 'e2e-inf-a-759b04-openshift-default', 'weight': 100}, 'wildcardPolicy': 'None'}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:301: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1130: in bootstrap_aitenant_tenant apply_gateway_route_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) test/e2e/tests/multitenancy_helpers.py:1013: in apply_gateway_route_fixture wait_for_route_admitted(route_name) test/e2e/tests/multitenancy_helpers.py:812: in wait_for_route_admitted return wait_for_json("route", route_name, GATEWAY_NAMESPACE, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'route', name = 'e2e-inf-a-759b04-route', namespace = 'openshift-ingress' predicate = <function wait_for_route_admitted.<locals>._predicate at 0x7fd0b24d8f40> timeout = 60, interval = 3 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: route/e2e-inf-a-759b04-route in openshift-ingress did not satisfy condition. Last object: {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"gateway","app.kubernetes.io/instance":"e2e-inf-a-759b04","app.kubernetes.io/name":"maas","e2e.maas.opendatahub.io/fixture":"e2e-inf-a-759b04","gateway.networking.k8s.io/gateway-name":"e2e-inf-a-759b04"},"name":"e2e-inf-a-759b04-route","namespace":"openshift-ingress"},"spec":{"host":"e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev","port":{"targetPort":443},"tls":{"destinationCACertificate":"-----BEGIN CERTIFICATE-----\\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\\n-----END CERTIFICATE-----\\n","insecureEdgeTerminationPolicy":"Redirect","termination":"reencrypt"},"to":{"kind":"Service","name":"e2e-inf-a-759b04-openshift-default","weight":100}}}\n'}, 'creationTimestamp': '2026-11-03T20:26:27Z', 'generation': 1, 'labels': {'app.kubernetes.io/component': 'gateway', 'app.kubernetes.io/instance': 'e2e-inf-a-759b04', 'app.kubernetes.io/name': 'maas', 'e2e.maas.opendatahub.io/fixture': 'e2e-inf-a-759b04', 'gateway.networking.k8s.io/gateway-name': 'e2e-inf-a-759b04'}, 'name': 'e2e-inf-a-759b04-route', 'namespace': 'openshift-ingress', 'resourceVersion': '157101', 'uid': '9b60ffb8-33dc-44d8-95d2-f167cad41f26'}, 'spec': {'host': 'e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', 'port': {'targetPort': 443}, 'tls': {'destinationCACertificate': '-----BEGIN CERTIFICATE-----\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\n-----END CERTIFICATE-----\n', 'insecureEdgeTerminationPolicy': 'Redirect', 'termination': 'reencrypt'}, 'to': {'kind': 'Service', 'name': 'e2e-inf-a-759b04-openshift-default', 'weight': 100}, 'wildcardPolicy': 'None'}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:301: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1130: in bootstrap_aitenant_tenant apply_gateway_route_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) test/e2e/tests/multitenancy_helpers.py:1013: in apply_gateway_route_fixture wait_for_route_admitted(route_name) test/e2e/tests/multitenancy_helpers.py:812: in wait_for_route_admitted return wait_for_json("route", route_name, GATEWAY_NAMESPACE, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'route', name = 'e2e-inf-a-759b04-route', namespace = 'openshift-ingress' predicate = <function wait_for_route_admitted.<locals>._predicate at 0x7fd0b24d8f40> timeout = 60, interval = 3 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: route/e2e-inf-a-759b04-route in openshift-ingress did not satisfy condition. Last object: {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"gateway","app.kubernetes.io/instance":"e2e-inf-a-759b04","app.kubernetes.io/name":"maas","e2e.maas.opendatahub.io/fixture":"e2e-inf-a-759b04","gateway.networking.k8s.io/gateway-name":"e2e-inf-a-759b04"},"name":"e2e-inf-a-759b04-route","namespace":"openshift-ingress"},"spec":{"host":"e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev","port":{"targetPort":443},"tls":{"destinationCACertificate":"-----BEGIN CERTIFICATE-----\\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\\n-----END CERTIFICATE-----\\n","insecureEdgeTerminationPolicy":"Redirect","termination":"reencrypt"},"to":{"kind":"Service","name":"e2e-inf-a-759b04-openshift-default","weight":100}}}\n'}, 'creationTimestamp': '2026-11-03T20:26:27Z', 'generation': 1, 'labels': {'app.kubernetes.io/component': 'gateway', 'app.kubernetes.io/instance': 'e2e-inf-a-759b04', 'app.kubernetes.io/name': 'maas', 'e2e.maas.opendatahub.io/fixture': 'e2e-inf-a-759b04', 'gateway.networking.k8s.io/gateway-name': 'e2e-inf-a-759b04'}, 'name': 'e2e-inf-a-759b04-route', 'namespace': 'openshift-ingress', 'resourceVersion': '157101', 'uid': '9b60ffb8-33dc-44d8-95d2-f167cad41f26'}, 'spec': {'host': 'e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', 'port': {'targetPort': 443}, 'tls': {'destinationCACertificate': '-----BEGIN CERTIFICATE-----\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\n-----END CERTIFICATE-----\n', 'insecureEdgeTerminationPolicy': 'Redirect', 'termination': 'reencrypt'}, 'to': {'kind': 'Service', 'name': 'e2e-inf-a-759b04-openshift-default', 'weight': 100}, 'wildcardPolicy': 'None'}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:301: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1130: in bootstrap_aitenant_tenant apply_gateway_route_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) test/e2e/tests/multitenancy_helpers.py:1013: in apply_gateway_route_fixture wait_for_route_admitted(route_name) test/e2e/tests/multitenancy_helpers.py:812: in wait_for_route_admitted return wait_for_json("route", route_name, GATEWAY_NAMESPACE, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'route', name = 'e2e-inf-a-759b04-route', namespace = 'openshift-ingress' predicate = <function wait_for_route_admitted.<locals>._predicate at 0x7fd0b24d8f40> timeout = 60, interval = 3 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: route/e2e-inf-a-759b04-route in openshift-ingress did not satisfy condition. Last object: {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"gateway","app.kubernetes.io/instance":"e2e-inf-a-759b04","app.kubernetes.io/name":"maas","e2e.maas.opendatahub.io/fixture":"e2e-inf-a-759b04","gateway.networking.k8s.io/gateway-name":"e2e-inf-a-759b04"},"name":"e2e-inf-a-759b04-route","namespace":"openshift-ingress"},"spec":{"host":"e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev","port":{"targetPort":443},"tls":{"destinationCACertificate":"-----BEGIN CERTIFICATE-----\\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\\n-----END CERTIFICATE-----\\n","insecureEdgeTerminationPolicy":"Redirect","termination":"reencrypt"},"to":{"kind":"Service","name":"e2e-inf-a-759b04-openshift-default","weight":100}}}\n'}, 'creationTimestamp': '2026-11-03T20:26:27Z', 'generation': 1, 'labels': {'app.kubernetes.io/component': 'gateway', 'app.kubernetes.io/instance': 'e2e-inf-a-759b04', 'app.kubernetes.io/name': 'maas', 'e2e.maas.opendatahub.io/fixture': 'e2e-inf-a-759b04', 'gateway.networking.k8s.io/gateway-name': 'e2e-inf-a-759b04'}, 'name': 'e2e-inf-a-759b04-route', 'namespace': 'openshift-ingress', 'resourceVersion': '157101', 'uid': '9b60ffb8-33dc-44d8-95d2-f167cad41f26'}, 'spec': {'host': 'e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', 'port': {'targetPort': 443}, 'tls': {'destinationCACertificate': '-----BEGIN CERTIFICATE-----\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\n-----END CERTIFICATE-----\n', 'insecureEdgeTerminationPolicy': 'Redirect', 'termination': 'reencrypt'}, 'to': {'kind': 'Service', 'name': 'e2e-inf-a-759b04-openshift-default', 'weight': 100}, 'wildcardPolicy': 'None'}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:301: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1130: in bootstrap_aitenant_tenant apply_gateway_route_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) test/e2e/tests/multitenancy_helpers.py:1013: in apply_gateway_route_fixture wait_for_route_admitted(route_name) test/e2e/tests/multitenancy_helpers.py:812: in wait_for_route_admitted return wait_for_json("route", route_name, GATEWAY_NAMESPACE, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'route', name = 'e2e-inf-a-759b04-route', namespace = 'openshift-ingress' predicate = <function wait_for_route_admitted.<locals>._predicate at 0x7fd0b24d8f40> timeout = 60, interval = 3 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: route/e2e-inf-a-759b04-route in openshift-ingress did not satisfy condition. Last object: {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"gateway","app.kubernetes.io/instance":"e2e-inf-a-759b04","app.kubernetes.io/name":"maas","e2e.maas.opendatahub.io/fixture":"e2e-inf-a-759b04","gateway.networking.k8s.io/gateway-name":"e2e-inf-a-759b04"},"name":"e2e-inf-a-759b04-route","namespace":"openshift-ingress"},"spec":{"host":"e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev","port":{"targetPort":443},"tls":{"destinationCACertificate":"-----BEGIN CERTIFICATE-----\\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\\n-----END CERTIFICATE-----\\n","insecureEdgeTerminationPolicy":"Redirect","termination":"reencrypt"},"to":{"kind":"Service","name":"e2e-inf-a-759b04-openshift-default","weight":100}}}\n'}, 'creationTimestamp': '2026-11-03T20:26:27Z', 'generation': 1, 'labels': {'app.kubernetes.io/component': 'gateway', 'app.kubernetes.io/instance': 'e2e-inf-a-759b04', 'app.kubernetes.io/name': 'maas', 'e2e.maas.opendatahub.io/fixture': 'e2e-inf-a-759b04', 'gateway.networking.k8s.io/gateway-name': 'e2e-inf-a-759b04'}, 'name': 'e2e-inf-a-759b04-route', 'namespace': 'openshift-ingress', 'resourceVersion': '157101', 'uid': '9b60ffb8-33dc-44d8-95d2-f167cad41f26'}, 'spec': {'host': 'e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', 'port': {'targetPort': 443}, 'tls': {'destinationCACertificate': '-----BEGIN CERTIFICATE-----\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\n-----END CERTIFICATE-----\n', 'insecureEdgeTerminationPolicy': 'Redirect', 'termination': 'reencrypt'}, 'to': {'kind': 'Service', 'name': 'e2e-inf-a-759b04-openshift-default', 'weight': 100}, 'wildcardPolicy': 'None'}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:301: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1130: in bootstrap_aitenant_tenant apply_gateway_route_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) test/e2e/tests/multitenancy_helpers.py:1013: in apply_gateway_route_fixture wait_for_route_admitted(route_name) test/e2e/tests/multitenancy_helpers.py:812: in wait_for_route_admitted return wait_for_json("route", route_name, GATEWAY_NAMESPACE, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'route', name = 'e2e-inf-a-759b04-route', namespace = 'openshift-ingress' predicate = <function wait_for_route_admitted.<locals>._predicate at 0x7fd0b24d8f40> timeout = 60, interval = 3 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: route/e2e-inf-a-759b04-route in openshift-ingress did not satisfy condition. Last object: {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"gateway","app.kubernetes.io/instance":"e2e-inf-a-759b04","app.kubernetes.io/name":"maas","e2e.maas.opendatahub.io/fixture":"e2e-inf-a-759b04","gateway.networking.k8s.io/gateway-name":"e2e-inf-a-759b04"},"name":"e2e-inf-a-759b04-route","namespace":"openshift-ingress"},"spec":{"host":"e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev","port":{"targetPort":443},"tls":{"destinationCACertificate":"-----BEGIN CERTIFICATE-----\\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\\n-----END CERTIFICATE-----\\n","insecureEdgeTerminationPolicy":"Redirect","termination":"reencrypt"},"to":{"kind":"Service","name":"e2e-inf-a-759b04-openshift-default","weight":100}}}\n'}, 'creationTimestamp': '2026-11-03T20:26:27Z', 'generation': 1, 'labels': {'app.kubernetes.io/component': 'gateway', 'app.kubernetes.io/instance': 'e2e-inf-a-759b04', 'app.kubernetes.io/name': 'maas', 'e2e.maas.opendatahub.io/fixture': 'e2e-inf-a-759b04', 'gateway.networking.k8s.io/gateway-name': 'e2e-inf-a-759b04'}, 'name': 'e2e-inf-a-759b04-route', 'namespace': 'openshift-ingress', 'resourceVersion': '157101', 'uid': '9b60ffb8-33dc-44d8-95d2-f167cad41f26'}, 'spec': {'host': 'e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', 'port': {'targetPort': 443}, 'tls': {'destinationCACertificate': '-----BEGIN CERTIFICATE-----\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\n-----END CERTIFICATE-----\n', 'insecureEdgeTerminationPolicy': 'Redirect', 'termination': 'reencrypt'}, 'to': {'kind': 'Service', 'name': 'e2e-inf-a-759b04-openshift-default', 'weight': 100}, 'wildcardPolicy': 'None'}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:301: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1130: in bootstrap_aitenant_tenant apply_gateway_route_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) test/e2e/tests/multitenancy_helpers.py:1013: in apply_gateway_route_fixture wait_for_route_admitted(route_name) test/e2e/tests/multitenancy_helpers.py:812: in wait_for_route_admitted return wait_for_json("route", route_name, GATEWAY_NAMESPACE, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'route', name = 'e2e-inf-a-759b04-route', namespace = 'openshift-ingress' predicate = <function wait_for_route_admitted.<locals>._predicate at 0x7fd0b24d8f40> timeout = 60, interval = 3 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: route/e2e-inf-a-759b04-route in openshift-ingress did not satisfy condition. Last object: {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"gateway","app.kubernetes.io/instance":"e2e-inf-a-759b04","app.kubernetes.io/name":"maas","e2e.maas.opendatahub.io/fixture":"e2e-inf-a-759b04","gateway.networking.k8s.io/gateway-name":"e2e-inf-a-759b04"},"name":"e2e-inf-a-759b04-route","namespace":"openshift-ingress"},"spec":{"host":"e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev","port":{"targetPort":443},"tls":{"destinationCACertificate":"-----BEGIN CERTIFICATE-----\\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\\n-----END CERTIFICATE-----\\n","insecureEdgeTerminationPolicy":"Redirect","termination":"reencrypt"},"to":{"kind":"Service","name":"e2e-inf-a-759b04-openshift-default","weight":100}}}\n'}, 'creationTimestamp': '2026-11-03T20:26:27Z', 'generation': 1, 'labels': {'app.kubernetes.io/component': 'gateway', 'app.kubernetes.io/instance': 'e2e-inf-a-759b04', 'app.kubernetes.io/name': 'maas', 'e2e.maas.opendatahub.io/fixture': 'e2e-inf-a-759b04', 'gateway.networking.k8s.io/gateway-name': 'e2e-inf-a-759b04'}, 'name': 'e2e-inf-a-759b04-route', 'namespace': 'openshift-ingress', 'resourceVersion': '157101', 'uid': '9b60ffb8-33dc-44d8-95d2-f167cad41f26'}, 'spec': {'host': 'e2e-inf-a-759b04.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', 'port': {'targetPort': 443}, 'tls': {'destinationCACertificate': '-----BEGIN CERTIFICATE-----\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\n-----END CERTIFICATE-----\n', 'insecureEdgeTerminationPolicy': 'Redirect', 'termination': 'reencrypt'}, 'to': {'kind': 'Service', 'name': 'e2e-inf-a-759b04-openshift-default', 'weight': 100}, 'wildcardPolicy': 'None'}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:301: AssertionErrorself = <test_tenant_namespace_discovery.TestTenantNamespaceDiscovery object at 0x7fd0b25f3710> def test_labeled_tenant_namespace_is_discovered(self): """1.1: Controller discovers labeled tenant namespaces and reconciles CRs.""" case = new_discovery_case(use_default_gateway=True) try: apply_discovery_labels(case["tenant_ns"], case["tenant_label_name"]) apply_tenant_cr(case["tenant_ns"], DEFAULT_GATEWAY_NAME) apply_maas_auth_policy(case["policy_name"], case["tenant_ns"]) apply_maas_subscription(case["subscription_name"], case["tenant_ns"]) wait_for_finalizer("maasauthpolicy", case["policy_name"], case["tenant_ns"], FINALIZER_AUTHPOLICY) wait_for_finalizer("maassubscription", case["subscription_name"], case["tenant_ns"], FINALIZER_SUBSCRIPTION) > auth = wait_for_status_phase( "maasauthpolicy", case["policy_name"], case["tenant_ns"], expected_phase="Active", ) test/e2e/tests/test_tenant_namespace_discovery.py:92: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:356: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'e2e-policy-625f4041' namespace = 'ai-tenant-e2e-mt-625f4041' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7fd0b23ca980> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/e2e-policy-625f4041 in ai-tenant-e2e-mt-625f4041 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"e2e-policy-625f4041","namespace":"ai-tenant-e2e-mt-625f4041"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-11-03T20:27:34Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'e2e-policy-625f4041', 'namespace': 'ai-tenant-e2e-mt-625f4041', 'resourceVersion': '170821', 'uid': 'abf26038-36f3-48e0-9f6e-543cb490ac9d'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-11-03T20:27:34Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/maas-gateway-auth to be accepted and enforced: AuthPolicy status observed generation 32, current generation is 77', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/e2e/tests/multitenancy_helpers.py:301: AssertionErrorself = <test_tenant_external_model_shared.TestSharedModelTwoGateways object at 0x7fb5a138d8d0> shared_env = {'tenant_a': {'suffix': '8b7351', 'tenant_ns': 'ai-tenant-e2e-extmod-a-8b7351', 'tenant_label_name': 'e2e-extmod-a-8b7...d-shr-user-324935', 'resources': <tenant_external_model_helpers.TenantExternalModelResources object at 0x7fb5a10e1910>} shared_model = {'model': {'name': 'e2e-shr-shared-7cb2ed', 'namespace': 'e2e-extmod-shr-user-324935', 'client_name': 'e2e-shr-shared-...'e2e-shr-shared-7cb2ed-e2e-extmod-a-8b7351', 'e2e-extmod-b-0e4ce7': 'e2e-shr-shared-7cb2ed-e2e-extmod-b-0e4ce7'}, ...}} @pytest.fixture(scope="class") def shared_serving(self, shared_env, shared_model): """Per-tenant auth/subscription stacks and API keys for the shared model.""" env = shared_env model = shared_model["model"] _require_maas_route(model) access_a = _serve_in_tenant(env, env["tenant_a"], model) > access_b = _serve_in_tenant(env, env["tenant_b"], model) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ test/e2e/tests/test_tenant_external_model_shared.py:214: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/test_tenant_external_model_shared.py:124: in _serve_in_tenant access = serve_external_model_in_tenant( test/e2e/tests/tenant_external_model_helpers.py:1265: in serve_external_model_in_tenant api_key = mint_tenant_api_key( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ case = {'suffix': '0e4ce7', 'tenant_ns': 'ai-tenant-e2e-extmod-b-0e4ce7', 'tenant_label_name': 'e2e-extmod-b-0e4ce7', 'gateway_name': 'e2e-extmod-b-0e4ce7', ...} key_name = 'e2e-extmod-2394068d' subscription_name = 'e2e-shr-shared-7cb2ed-sub' def mint_tenant_api_key(case: dict[str, str], key_name: str, subscription_name: str) -> str: response = create_api_key_at(case["maas_api_url"], _get_cluster_token(), key_name, subscription=subscription_name) > assert response.status_code in (200, 201), ( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ f"failed to mint API key on tenant {case['tenant_label_name']} maas-api: " f"{response.status_code} {redact_sensitive(response.text)}" ) E AssertionError: failed to mint API key on tenant e2e-extmod-b-0e4ce7 maas-api: 504 <html><body><h1>504 Gateway Time-out</h1> E The server didn't respond in time. E </body></html> test/e2e/tests/tenant_external_model_helpers.py:1278: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1130: in bootstrap_aitenant_tenant apply_gateway_route_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) test/e2e/tests/multitenancy_helpers.py:1013: in apply_gateway_route_fixture wait_for_route_admitted(route_name) test/e2e/tests/multitenancy_helpers.py:812: in wait_for_route_admitted return wait_for_json("route", route_name, GATEWAY_NAMESPACE, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'route', name = 'e2e-inf-a-28b78c-route', namespace = 'openshift-ingress' predicate = <function wait_for_route_admitted.<locals>._predicate at 0x7f52fdb0a340> timeout = 60, interval = 3 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: route/e2e-inf-a-28b78c-route in openshift-ingress did not satisfy condition. Last object: {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"gateway","app.kubernetes.io/instance":"e2e-inf-a-28b78c","app.kubernetes.io/name":"maas","e2e.maas.opendatahub.io/fixture":"e2e-inf-a-28b78c","gateway.networking.k8s.io/gateway-name":"e2e-inf-a-28b78c"},"name":"e2e-inf-a-28b78c-route","namespace":"openshift-ingress"},"spec":{"host":"e2e-inf-a-28b78c.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev","port":{"targetPort":443},"tls":{"destinationCACertificate":"-----BEGIN CERTIFICATE-----\\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\\n-----END CERTIFICATE-----\\n","insecureEdgeTerminationPolicy":"Redirect","termination":"reencrypt"},"to":{"kind":"Service","name":"e2e-inf-a-28b78c-openshift-default","weight":100}}}\n'}, 'creationTimestamp': '2026-11-03T20:26:48Z', 'generation': 1, 'labels': {'app.kubernetes.io/component': 'gateway', 'app.kubernetes.io/instance': 'e2e-inf-a-28b78c', 'app.kubernetes.io/name': 'maas', 'e2e.maas.opendatahub.io/fixture': 'e2e-inf-a-28b78c', 'gateway.networking.k8s.io/gateway-name': 'e2e-inf-a-28b78c'}, 'name': 'e2e-inf-a-28b78c-route', 'namespace': 'openshift-ingress', 'resourceVersion': '158041', 'uid': '152d6bce-6bb7-4af2-8035-65b609e4c3d9'}, 'spec': {'host': 'e2e-inf-a-28b78c.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', 'port': {'targetPort': 443}, 'tls': {'destinationCACertificate': '-----BEGIN CERTIFICATE-----\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\n-----END CERTIFICATE-----\n', 'insecureEdgeTerminationPolicy': 'Redirect', 'termination': 'reencrypt'}, 'to': {'kind': 'Service', 'name': 'e2e-inf-a-28b78c-openshift-default', 'weight': 100}, 'wildcardPolicy': 'None'}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:301: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1130: in bootstrap_aitenant_tenant apply_gateway_route_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) test/e2e/tests/multitenancy_helpers.py:1013: in apply_gateway_route_fixture wait_for_route_admitted(route_name) test/e2e/tests/multitenancy_helpers.py:812: in wait_for_route_admitted return wait_for_json("route", route_name, GATEWAY_NAMESPACE, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'route', name = 'e2e-inf-a-28b78c-route', namespace = 'openshift-ingress' predicate = <function wait_for_route_admitted.<locals>._predicate at 0x7f52fdb0a340> timeout = 60, interval = 3 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: route/e2e-inf-a-28b78c-route in openshift-ingress did not satisfy condition. Last object: {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"gateway","app.kubernetes.io/instance":"e2e-inf-a-28b78c","app.kubernetes.io/name":"maas","e2e.maas.opendatahub.io/fixture":"e2e-inf-a-28b78c","gateway.networking.k8s.io/gateway-name":"e2e-inf-a-28b78c"},"name":"e2e-inf-a-28b78c-route","namespace":"openshift-ingress"},"spec":{"host":"e2e-inf-a-28b78c.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev","port":{"targetPort":443},"tls":{"destinationCACertificate":"-----BEGIN CERTIFICATE-----\\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\\n-----END CERTIFICATE-----\\n","insecureEdgeTerminationPolicy":"Redirect","termination":"reencrypt"},"to":{"kind":"Service","name":"e2e-inf-a-28b78c-openshift-default","weight":100}}}\n'}, 'creationTimestamp': '2026-11-03T20:26:48Z', 'generation': 1, 'labels': {'app.kubernetes.io/component': 'gateway', 'app.kubernetes.io/instance': 'e2e-inf-a-28b78c', 'app.kubernetes.io/name': 'maas', 'e2e.maas.opendatahub.io/fixture': 'e2e-inf-a-28b78c', 'gateway.networking.k8s.io/gateway-name': 'e2e-inf-a-28b78c'}, 'name': 'e2e-inf-a-28b78c-route', 'namespace': 'openshift-ingress', 'resourceVersion': '158041', 'uid': '152d6bce-6bb7-4af2-8035-65b609e4c3d9'}, 'spec': {'host': 'e2e-inf-a-28b78c.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', 'port': {'targetPort': 443}, 'tls': {'destinationCACertificate': '-----BEGIN CERTIFICATE-----\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\n-----END CERTIFICATE-----\n', 'insecureEdgeTerminationPolicy': 'Redirect', 'termination': 'reencrypt'}, 'to': {'kind': 'Service', 'name': 'e2e-inf-a-28b78c-openshift-default', 'weight': 100}, 'wildcardPolicy': 'None'}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:301: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1132: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-d2aa3b', namespace = 'ai-tenants' predicate = <function aitenant_ready at 0x7f9f94c43f60>, timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-inf-a-d2aa3b in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-inf-a-d2aa3b","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-inf-a-d2aa3b"}}}\n'}, 'creationTimestamp': '2026-11-03T20:25:19Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-inf-a-d2aa3b', 'namespace': 'ai-tenants', 'resourceVersion': '154596', 'uid': 'd6544451-efc3-40d0-ba22-90e066cb6292'}, 'spec': {'gateway': {'name': 'e2e-inf-a-d2aa3b'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-11-03T20:25:20Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-inf-a-d2aa3b', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-inf-a-d2aa3b'}} test/e2e/tests/multitenancy_helpers.py:301: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1132: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-d2aa3b', namespace = 'ai-tenants' predicate = <function aitenant_ready at 0x7f9f94c43f60>, timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-inf-a-d2aa3b in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-inf-a-d2aa3b","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-inf-a-d2aa3b"}}}\n'}, 'creationTimestamp': '2026-11-03T20:25:19Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-inf-a-d2aa3b', 'namespace': 'ai-tenants', 'resourceVersion': '154596', 'uid': 'd6544451-efc3-40d0-ba22-90e066cb6292'}, 'spec': {'gateway': {'name': 'e2e-inf-a-d2aa3b'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-11-03T20:25:20Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-inf-a-d2aa3b', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-inf-a-d2aa3b'}} test/e2e/tests/multitenancy_helpers.py:301: AssertionErrorself = <test_tenant_namespace_discovery.TestTenantNamespaceDiscovery object at 0x7f9f9482a010> def test_tenant_admin_rbac_is_namespace_scoped(self): """1.6: Tenant-admin Role from AITenant bootstrap is scoped to the tenant namespace.""" case = new_discovery_case() sa_name = f"e2e-mt-rbac-{case['suffix']}" role_name = f"aitenant-{case['tenant_label_name']}-tenant-admin" other_ns = f"e2e-mt-other-{case['suffix']}" try: apply_gateway_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_namespace_discovery.py:260: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1130: in bootstrap_aitenant_tenant apply_gateway_route_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) test/e2e/tests/multitenancy_helpers.py:1013: in apply_gateway_route_fixture wait_for_route_admitted(route_name) test/e2e/tests/multitenancy_helpers.py:812: in wait_for_route_admitted return wait_for_json("route", route_name, GATEWAY_NAMESPACE, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'route', name = 'e2e-mt-96230c5b-route', namespace = 'openshift-ingress' predicate = <function wait_for_route_admitted.<locals>._predicate at 0x7f9f94617e20> timeout = 60, interval = 3 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: route/e2e-mt-96230c5b-route in openshift-ingress did not satisfy condition. Last object: {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"gateway","app.kubernetes.io/instance":"e2e-mt-96230c5b","app.kubernetes.io/name":"maas","e2e.maas.opendatahub.io/fixture":"e2e-mt-96230c5b","gateway.networking.k8s.io/gateway-name":"e2e-mt-96230c5b"},"name":"e2e-mt-96230c5b-route","namespace":"openshift-ingress"},"spec":{"host":"e2e-mt-96230c5b.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev","port":{"targetPort":443},"tls":{"destinationCACertificate":"-----BEGIN CERTIFICATE-----\\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\\n-----END CERTIFICATE-----\\n","insecureEdgeTerminationPolicy":"Redirect","termination":"reencrypt"},"to":{"kind":"Service","name":"e2e-mt-96230c5b-openshift-default","weight":100}}}\n'}, 'creationTimestamp': '2026-11-03T20:29:46Z', 'generation': 1, 'labels': {'app.kubernetes.io/component': 'gateway', 'app.kubernetes.io/instance': 'e2e-mt-96230c5b', 'app.kubernetes.io/name': 'maas', 'e2e.maas.opendatahub.io/fixture': 'e2e-mt-96230c5b', 'gateway.networking.k8s.io/gateway-name': 'e2e-mt-96230c5b'}, 'name': 'e2e-mt-96230c5b-route', 'namespace': 'openshift-ingress', 'resourceVersion': '167431', 'uid': '726c4f62-9714-4ea0-b1b2-dcdb57b3121d'}, 'spec': {'host': 'e2e-mt-96230c5b.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', 'port': {'targetPort': 443}, 'tls': {'destinationCACertificate': '-----BEGIN CERTIFICATE-----\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\n-----END CERTIFICATE-----\n', 'insecureEdgeTerminationPolicy': 'Redirect', 'termination': 'reencrypt'}, 'to': {'kind': 'Service', 'name': 'e2e-mt-96230c5b-openshift-default', 'weight': 100}, 'wildcardPolicy': 'None'}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:301: AssertionErrorself = <test_tenant_namespace_discovery.TestTenantNamespaceDiscovery object at 0x7f52fdc352d0> def test_dynamic_discovery_after_label_added(self): """1.2 variant: Pre-existing CRs reconcile after namespace gains discovery labels.""" case = new_discovery_case(use_default_gateway=True) try: ensure_namespace(case["tenant_ns"]) apply_tenant_cr(case["tenant_ns"], DEFAULT_GATEWAY_NAME) apply_maas_auth_policy(case["policy_name"], case["tenant_ns"]) time.sleep(10) before = get_json_or_none("maasauthpolicy", case["policy_name"], case["tenant_ns"]) assert before is not None assert FINALIZER_AUTHPOLICY not in ((before.get("metadata") or {}).get("finalizers") or []) apply_discovery_labels(case["tenant_ns"], case["tenant_label_name"]) wait_for_finalizer("maasauthpolicy", case["policy_name"], case["tenant_ns"], FINALIZER_AUTHPOLICY) > wait_for_status_phase( "maasauthpolicy", case["policy_name"], case["tenant_ns"], expected_phase="Active", ) test/e2e/tests/test_tenant_namespace_discovery.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:356: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'e2e-policy-c5c8526c' namespace = 'ai-tenant-e2e-mt-c5c8526c' predicate = <function wait_for_status_phase.<locals>._predicate at 0x7f52fdb8b2e0> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/e2e-policy-c5c8526c in ai-tenant-e2e-mt-c5c8526c did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"e2e-policy-c5c8526c","namespace":"ai-tenant-e2e-mt-c5c8526c"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-11-03T20:29:41Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'e2e-policy-c5c8526c', 'namespace': 'ai-tenant-e2e-mt-c5c8526c', 'resourceVersion': '181318', 'uid': 'a0ebd651-bdd7-4cad-a499-dbb7bffd1eb5'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-11-03T20:29:52Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/maas-gateway-auth to be accepted and enforced: AuthPolicy status observed generation 32, current generation is 111', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/e2e/tests/multitenancy_helpers.py:301: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:1130: in bootstrap_aitenant_tenant apply_gateway_route_fixture(case["gateway_name"], fixture_label=case["tenant_label_name"]) test/e2e/tests/multitenancy_helpers.py:1013: in apply_gateway_route_fixture wait_for_route_admitted(route_name) test/e2e/tests/multitenancy_helpers.py:812: in wait_for_route_admitted return wait_for_json("route", route_name, GATEWAY_NAMESPACE, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'route', name = 'e2e-inf-a-086365-route', namespace = 'openshift-ingress' predicate = <function wait_for_route_admitted.<locals>._predicate at 0x7fb5a10a4fe0> timeout = 60, interval = 3 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: route/e2e-inf-a-086365-route in openshift-ingress did not satisfy condition. Last object: {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"gateway","app.kubernetes.io/instance":"e2e-inf-a-086365","app.kubernetes.io/name":"maas","e2e.maas.opendatahub.io/fixture":"e2e-inf-a-086365","gateway.networking.k8s.io/gateway-name":"e2e-inf-a-086365"},"name":"e2e-inf-a-086365-route","namespace":"openshift-ingress"},"spec":{"host":"e2e-inf-a-086365.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev","port":{"targetPort":443},"tls":{"destinationCACertificate":"-----BEGIN CERTIFICATE-----\\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\\n-----END CERTIFICATE-----\\n","insecureEdgeTerminationPolicy":"Redirect","termination":"reencrypt"},"to":{"kind":"Service","name":"e2e-inf-a-086365-openshift-default","weight":100}}}\n'}, 'creationTimestamp': '2026-11-03T20:28:38Z', 'generation': 1, 'labels': {'app.kubernetes.io/component': 'gateway', 'app.kubernetes.io/instance': 'e2e-inf-a-086365', 'app.kubernetes.io/name': 'maas', 'e2e.maas.opendatahub.io/fixture': 'e2e-inf-a-086365', 'gateway.networking.k8s.io/gateway-name': 'e2e-inf-a-086365'}, 'name': 'e2e-inf-a-086365-route', 'namespace': 'openshift-ingress', 'resourceVersion': '164288', 'uid': 'edf874a5-77a4-4a9a-8543-c2ce3a3490da'}, 'spec': {'host': 'e2e-inf-a-086365.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', 'port': {'targetPort': 443}, 'tls': {'destinationCACertificate': '-----BEGIN CERTIFICATE-----\nMIIDUTCCAjmgAwIBAgIIPa8XXlMYkUowDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE\nAwwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTc5MzczNDYwMDAe\nFw0yNjExMDMxOTM2MzlaFw0yOTAxMDExOTM2NDBaMDYxNDAyBgNVBAMMK29wZW5z\naGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3OTM3MzQ2MDAwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuESLdcYS88PpkvZOaIx34dTGVFzqUE6ne\nJDyMeQn+qRdX3Qe7t4Wosr12WRyZifg1/TYGPYg5TUef8uzKb4HfhJ/3oGaBsgMx\nwtFaFZvvGc0aiyH1UcGZQ9/PlRK4F/UnibUAPmTuxD7HQtno7BLsRry8QxxI5C29\nb2niiS5EdFGQub5dA8ioxd3grvg+/rKX79i+kUntjVKFoPkCKDkkFwsK7C09Po/y\nhApRNRXMz8ObwuTyokSMSGNociAEz8XcBK3+7GjGUVarnxrAohxwoNlHS88cSuSb\nBNB/9M98a8qyH+193vFZEhy15A+/m5Saj+Ph4V2qmv0ygzFWnCzPAgMBAAGjYzBh\nMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUqiJT\ndLJVySmvrXK8RsmJGMCShTAfBgNVHSMEGDAWgBRUqiJTdLJVySmvrXK8RsmJGMCS\nhTANBgkqhkiG9w0BAQsFAAOCAQEAIcXcWwCWRnvyYnVfUiIvOiWBIZISzXdVEwUX\nTTte8+QqM3Ur9WfmHCU3YV1q/t041E1tHFR8QdkYUU9vy7CfKH9HHAZJdJZEl9bv\nizMqeIYagfDbiKkOpD71cjo5ZtzC42xhrHSrH3otFEZochms6xAUP+MB98stNWVm\n62GCZfYyKCvPLUh1wJNRcHRoyYan6+IDjT2rhoWTeOUO6vElJ0lrMOf+MV4WK6UE\n79AN1Bt/Rc1NiKZUAy1gjwmAsLrtJbyxgeh/lnpCVjz1XDL/gGyMODKvN5NGGWTw\n8PsEqdZWGQoK6sm4ewR6MyE9/Jv8FYtBd9r+R6w+yMEpY5O+Pw==\n-----END CERTIFICATE-----\n', 'insecureEdgeTerminationPolicy': 'Redirect', 'termination': 'reencrypt'}, 'to': {'kind': 'Service', 'name': 'e2e-inf-a-086365-openshift-default', 'weight': 100}, 'wildcardPolicy': 'None'}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:301: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) # Store model path (with /v1 for OpenAI API compatibility) case["model_path"] = f"/{case['tenant_ns']}/{model_name}/v1" for case in (case_a, case_b): > wait_for_route_auth_enforced( f"{_get_tenant_gateway_url(case['gateway_name'])}{case['model_path']}" ) test/e2e/tests/test_tenant_model_inference.py:189: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:443: in wait_for_route_auth_enforced wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function wait_for_route_auth_enforced.<locals>._check at 0x7fa60fbd6200> timeout = 180 what = 'https://e2e-inf-a-e7b70d.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev/ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d/v1/chat/completions did not reject an unknown API key for 10s' interval = 2 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: https://e2e-inf-a-e7b70d.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev/ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d/v1/chat/completions did not reject an unknown API key for 10s within 180s; last: HTTPSConnectionPool(host='e2e-inf-a-e7b70d.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', port=443): Max retries exceeded with url: /ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d/v1/chat/completions (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1004)'))) test/e2e/tests/multitenancy_helpers.py:279: AssertionError@pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): bootstrap_aitenant_tenant(case) # Create both LLMInferenceServices before waiting on either. The llmisvc # controller reconciles them one at a time and runs minutes behind under # parallel load, so B should queue while A's wait runs, not after it. for case in (case_a, case_b): # Track model name early for cleanup case["model_name"] = f"test-model-{case['suffix']}" _create_llmis( case["model_name"], case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE, ) for case in (case_a, case_b): model_name = case["model_name"] # Re-applies the LLMIS created above unchanged, then waits for it. provision_tenant_model( model_name, case["tenant_ns"], case["gateway_name"], ready_timeout=MODEL_BACKEND_READY_TIMEOUT, ) make_tenant_model_accessible( model_name, case["tenant_ns"], f"{model_name}-auth", f"{model_name}-sub", token_limit=10000, gateway_name=case["gateway_name"], ) # Store model path (with /v1 for OpenAI API compatibility) case["model_path"] = f"/{case['tenant_ns']}/{model_name}/v1" for case in (case_a, case_b): > wait_for_route_auth_enforced( f"{_get_tenant_gateway_url(case['gateway_name'])}{case['model_path']}" ) test/e2e/tests/test_tenant_model_inference.py:189: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:443: in wait_for_route_auth_enforced wait_until( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ check = <function wait_for_route_auth_enforced.<locals>._check at 0x7fa60fbd6200> timeout = 180 what = 'https://e2e-inf-a-e7b70d.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev/ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d/v1/chat/completions did not reject an unknown API key for 10s' interval = 2 def wait_until(check, timeout: int, what: str, interval: int = 5): """Poll check() until it returns a truthy value. Assertion failures and transient cluster or HTTP errors count as "not yet"; the last one ends up in the timeout message.""" deadline = time.time() + timeout while True: try: result = check() if result: return result last = result except (AssertionError, subprocess.TimeoutExpired, requests.RequestException) as exc: last = exc > assert time.time() < deadline, f"{what} within {timeout}s; last: {last}" ^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: https://e2e-inf-a-e7b70d.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev/ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d/v1/chat/completions did not reject an unknown API key for 10s within 180s; last: HTTPSConnectionPool(host='e2e-inf-a-e7b70d.apps.00125a180bb08b60980b.prod.konflux-ocp-ci.dev', port=443): Max retries exceeded with url: /ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d/v1/chat/completions (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1004)'))) test/e2e/tests/multitenancy_helpers.py:279: AssertionError