/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_external_oidc.py:252: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:263: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:277: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:290: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:303: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:335: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:379: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:395: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:405: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:428: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:436: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:441: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:454: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:501: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:537: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:602: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:633: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:659: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:690: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:750: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:786: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:824: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:872: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:961: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1013: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1049: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1103: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1142: EXTERNAL_OIDC not enabledself = <test_model_identity_conflict.TestModelIdentityConflictDetection object at 0x7f1875869100> worker_tenant_context = WorkerTenantContext(worker_id='w4', suffix='651569', tenant_name='e2e-worker-w4-651569', tenant_namespace='ai-tenant-e...ulator-access', subscription_name='simulator-subscription', premium_subscription_name='premium-simulator-subscription') def test_colliding_model_names_flagged_then_resolved(self, worker_tenant_context): model_namespace = worker_tenant_context.model_namespace gateway_name = worker_tenant_context.gateway_name suffix = uuid.uuid4().hex[:8] shared_model_name = f"test/e2e-identity-conflict-{suffix}" llmis_a = f"e2e-conflict-a-{suffix}" llmis_b = f"e2e-conflict-b-{suffix}" ref_a, ref_b = llmis_a, llmis_b try: # A single model with no siblings must be reported unique. _create_llmis(llmis_a, model_namespace, gateway_name, GATEWAY_NAMESPACE, model_name=shared_model_name) > wait_for_llmisvc_backend_ready(llmis_a, model_namespace, gateway_name) test/e2e/tests/test_model_identity_conflict.py:88: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'e2e-conflict-a-e40ebf73' namespace = 'e2e-models-e2e-worker-w4-651569' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/e2e-conflict-a-e40ebf73 in e2e-models-e2e-worker-w4-651569 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"e2e-conflict-a-e40ebf73","namespace":"e2e-models-e2e-worker-w4-651569"},"spec":{"model":{"name":"test/e2e-identity-conflict-e40ebf73","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w4-651569","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","test/e2e-identity-conflict-e40ebf73","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T01:14:01Z', 'generation': 1, 'name': 'e2e-conflict-a-e40ebf73', 'namespace': 'e2e-models-e2e-worker-w4-651569', 'resourceVersion': '33591', 'uid': '96667edc-a49c-4bd1-85cc-4bf8e2edb873'}, 'spec': {'model': {'name': 'test/e2e-identity-conflict-e40ebf73', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w4-651569', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'test/e2e-identity-conflict-e40ebf73', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:229: AssertionError/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:189: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deploy/workspace/source/test/e2e/tests/test_api_keys.py:1470: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:355: Dormant-mode test mutates controller flags; set ENABLE_TENANT_DISCOVERY_DORMANT_E2E=trueself = <test_tenant_auto_resolve.TestTenantAutoResolve object at 0x7f4594599bb0> def test_no_matching_tenant_enters_failed(self): """MaaSModelRef enters Failed state when no AITenant matches the gateway.""" case = new_named_tenant_case("e2e-noten") model_name = f"noten-model-{case['suffix']}" fixture_label = f"noten-{case['suffix']}" try: apply_gateway_fixture(case["gateway_name"], fixture_label=fixture_label) wait_for_gateway_programmed(case["gateway_name"]) from multitenancy_helpers import ensure_namespace ensure_namespace(case["tenant_ns"]) apply_gateway_access_label(case["tenant_ns"], case["gateway_name"]) _create_llmis(model_name, case["tenant_ns"], case["gateway_name"], GATEWAY_NAMESPACE) > wait_for_httproute_accepted( f"{model_name}-kserve-route", case["tenant_ns"], case["gateway_name"], timeout=180, ) test/e2e/tests/test_tenant_auto_resolve.py:174: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:694: in wait_for_httproute_accepted return wait_for_json("httproute", route_name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'httproute', name = 'noten-model-836d37-kserve-route' namespace = 'ai-tenant-e2e-noten-836d37' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: httproute/noten-model-836d37-kserve-route in ai-tenant-e2e-noten-836d37 did not satisfy condition. Last object: None test/e2e/tests/multitenancy_helpers.py:229: AssertionError/workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners.self = <test_multi_tenant_integration.TestMultiTenantIntegration object at 0x7f7b515560a0> def test_full_tenant_lifecycle_create_to_delete(self): """7.1: Full tenant lifecycle from create through policy/subscription reconcile to delete.""" case = new_discovery_case() role_name = f"aitenant-{case['tenant_label_name']}-tenant-admin" try: bootstrap_aitenant_tenant(case) tenant = wait_for_json(TENANT_CONFIG_KIND, TENANT_CR_NAME, case["tenant_ns"], timeout=180) tenant_labels = tenant["metadata"].get("labels") or {} tenant_annotations = tenant["metadata"].get("annotations") or {} assert tenant_labels[LABEL_MANAGED_BY_AITENANT] == "true" assert tenant_labels[LABEL_TENANT_NAME] == case["tenant_label_name"] assert tenant_labels[LABEL_TENANT_NAMESPACE] == case["tenant_ns"] assert tenant_annotations[ANNOTATION_AITENANT_NAME] == case["tenant_label_name"] assert tenant_annotations[ANNOTATION_AITENANT_NAMESPACE] == AITENANT_NAMESPACE aitenant = wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, timeout=180) assert aitenant["status"]["gatewayRef"]["name"] == case["gateway_name"] assert get_json_or_none("role", role_name, case["tenant_ns"]) is not None model_name = f"e2e-lifecycle-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_multi_tenant_integration.py:94: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:941: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'e2e-lifecycle-model-194eb64c' namespace = 'ai-tenant-e2e-mt-194eb64c' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/e2e-lifecycle-model-194eb64c in ai-tenant-e2e-mt-194eb64c did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"e2e-lifecycle-model-194eb64c","namespace":"ai-tenant-e2e-mt-194eb64c"},"spec":{"model":{"name":"facebook/opt-125m","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-mt-194eb64c","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","facebook/opt-125m","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T01:17:09Z', 'generation': 1, 'name': 'e2e-lifecycle-model-194eb64c', 'namespace': 'ai-tenant-e2e-mt-194eb64c', 'resourceVersion': '38909', 'uid': '02787499-15c1-47d1-9799-7c688e8aa486'}, 'spec': {'model': {'name': 'facebook/opt-125m', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-mt-194eb64c', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'facebook/opt-125m', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_subscription_context' for <Function test_authorized_user_gets_200>> @pytest.fixture(scope="module", autouse=True) def _worker_subscription_context(request): """Point non-serial subscription tests at worker-owned tenant resources.""" from worker_tenant_fixtures import ( activate_worker_tenant, serial_only_selection, wait_for_worker_model_backends, ) if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > wait_for_worker_model_backends( context, ( context.distinct_model_ref, context.distinct_model_2_ref, context.unconfigured_model_ref, ), ) test/e2e/tests/test_subscription.py:121: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:300: in wait_for_worker_model_backends wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w0-6ba087' namespace = 'e2e-models-e2e-worker-w0-6ba087' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-w0-6ba087 in e2e-models-e2e-worker-w0-6ba087 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-w0-6ba087","namespace":"e2e-models-e2e-worker-w0-6ba087"},"spec":{"model":{"name":"e2e/distinct-w0-6ba087","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-6ba087","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-w0-6ba087","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T01:14:18Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-w0-6ba087', 'namespace': 'e2e-models-e2e-worker-w0-6ba087', 'resourceVersion': '39433', 'uid': '9c2bb4f5-66a7-4f89-9bf6-0909bc869b61'}, 'spec': {'model': {'name': 'e2e/distinct-w0-6ba087', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-w0-6ba087', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-6ba087/models/e2e/distinct-w0-6ba087'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-6ba087/models/e2e/distinct-w0-6ba087'}, {'name': 'e2e/distinct-w0-6ba087'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-w0-6ba087-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-6ba087'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-w0-6ba087-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-w0-6ba087-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_subscription_context' for <Function test_authorized_user_gets_200>> @pytest.fixture(scope="module", autouse=True) def _worker_subscription_context(request): """Point non-serial subscription tests at worker-owned tenant resources.""" from worker_tenant_fixtures import ( activate_worker_tenant, serial_only_selection, wait_for_worker_model_backends, ) if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > wait_for_worker_model_backends( context, ( context.distinct_model_ref, context.distinct_model_2_ref, context.unconfigured_model_ref, ), ) test/e2e/tests/test_subscription.py:121: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:300: in wait_for_worker_model_backends wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w0-6ba087' namespace = 'e2e-models-e2e-worker-w0-6ba087' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-w0-6ba087 in e2e-models-e2e-worker-w0-6ba087 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-w0-6ba087","namespace":"e2e-models-e2e-worker-w0-6ba087"},"spec":{"model":{"name":"e2e/distinct-w0-6ba087","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-6ba087","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-w0-6ba087","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T01:14:18Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-w0-6ba087', 'namespace': 'e2e-models-e2e-worker-w0-6ba087', 'resourceVersion': '39433', 'uid': '9c2bb4f5-66a7-4f89-9bf6-0909bc869b61'}, 'spec': {'model': {'name': 'e2e/distinct-w0-6ba087', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-w0-6ba087', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-6ba087/models/e2e/distinct-w0-6ba087'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-6ba087/models/e2e/distinct-w0-6ba087'}, {'name': 'e2e/distinct-w0-6ba087'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-w0-6ba087-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-6ba087'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-w0-6ba087-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-w0-6ba087-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_subscription_context' for <Function test_authorized_user_gets_200>> @pytest.fixture(scope="module", autouse=True) def _worker_subscription_context(request): """Point non-serial subscription tests at worker-owned tenant resources.""" from worker_tenant_fixtures import ( activate_worker_tenant, serial_only_selection, wait_for_worker_model_backends, ) if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > wait_for_worker_model_backends( context, ( context.distinct_model_ref, context.distinct_model_2_ref, context.unconfigured_model_ref, ), ) test/e2e/tests/test_subscription.py:121: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:300: in wait_for_worker_model_backends wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w0-6ba087' namespace = 'e2e-models-e2e-worker-w0-6ba087' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-w0-6ba087 in e2e-models-e2e-worker-w0-6ba087 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-w0-6ba087","namespace":"e2e-models-e2e-worker-w0-6ba087"},"spec":{"model":{"name":"e2e/distinct-w0-6ba087","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-6ba087","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-w0-6ba087","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T01:14:18Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-w0-6ba087', 'namespace': 'e2e-models-e2e-worker-w0-6ba087', 'resourceVersion': '39433', 'uid': '9c2bb4f5-66a7-4f89-9bf6-0909bc869b61'}, 'spec': {'model': {'name': 'e2e/distinct-w0-6ba087', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-w0-6ba087', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-6ba087/models/e2e/distinct-w0-6ba087'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-6ba087/models/e2e/distinct-w0-6ba087'}, {'name': 'e2e/distinct-w0-6ba087'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-w0-6ba087-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-6ba087'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-w0-6ba087-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-w0-6ba087-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_subscription_context' for <Function test_authorized_user_gets_200>> @pytest.fixture(scope="module", autouse=True) def _worker_subscription_context(request): """Point non-serial subscription tests at worker-owned tenant resources.""" from worker_tenant_fixtures import ( activate_worker_tenant, serial_only_selection, wait_for_worker_model_backends, ) if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > wait_for_worker_model_backends( context, ( context.distinct_model_ref, context.distinct_model_2_ref, context.unconfigured_model_ref, ), ) test/e2e/tests/test_subscription.py:121: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:300: in wait_for_worker_model_backends wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w0-6ba087' namespace = 'e2e-models-e2e-worker-w0-6ba087' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-w0-6ba087 in e2e-models-e2e-worker-w0-6ba087 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-w0-6ba087","namespace":"e2e-models-e2e-worker-w0-6ba087"},"spec":{"model":{"name":"e2e/distinct-w0-6ba087","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-6ba087","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-w0-6ba087","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T01:14:18Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-w0-6ba087', 'namespace': 'e2e-models-e2e-worker-w0-6ba087', 'resourceVersion': '39433', 'uid': '9c2bb4f5-66a7-4f89-9bf6-0909bc869b61'}, 'spec': {'model': {'name': 'e2e/distinct-w0-6ba087', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-w0-6ba087', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-6ba087/models/e2e/distinct-w0-6ba087'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-6ba087/models/e2e/distinct-w0-6ba087'}, {'name': 'e2e/distinct-w0-6ba087'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-w0-6ba087-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-6ba087'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-w0-6ba087-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-w0-6ba087-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_subscription_context' for <Function test_authorized_user_gets_200>> @pytest.fixture(scope="module", autouse=True) def _worker_subscription_context(request): """Point non-serial subscription tests at worker-owned tenant resources.""" from worker_tenant_fixtures import ( activate_worker_tenant, serial_only_selection, wait_for_worker_model_backends, ) if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > wait_for_worker_model_backends( context, ( context.distinct_model_ref, context.distinct_model_2_ref, context.unconfigured_model_ref, ), ) test/e2e/tests/test_subscription.py:121: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:300: in wait_for_worker_model_backends wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w0-6ba087' namespace = 'e2e-models-e2e-worker-w0-6ba087' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-w0-6ba087 in e2e-models-e2e-worker-w0-6ba087 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-w0-6ba087","namespace":"e2e-models-e2e-worker-w0-6ba087"},"spec":{"model":{"name":"e2e/distinct-w0-6ba087","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-6ba087","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-w0-6ba087","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T01:14:18Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-w0-6ba087', 'namespace': 'e2e-models-e2e-worker-w0-6ba087', 'resourceVersion': '39433', 'uid': '9c2bb4f5-66a7-4f89-9bf6-0909bc869b61'}, 'spec': {'model': {'name': 'e2e/distinct-w0-6ba087', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-w0-6ba087', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-6ba087/models/e2e/distinct-w0-6ba087'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-6ba087/models/e2e/distinct-w0-6ba087'}, {'name': 'e2e/distinct-w0-6ba087'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T01:17:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-w0-6ba087-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-6ba087'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-6ba087', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-6ba087-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-w0-6ba087-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-w0-6ba087-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionError/workspace/source/test/e2e/tests/test_gateway_scoped_authpolicy.py:219: legacy gateway-default-auth is scoped to the shared default gateway/workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:215: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B