/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_external_oidc.py:252: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:263: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:277: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:290: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:303: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:335: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:379: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:395: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:405: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:428: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:436: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:441: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:454: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:501: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:537: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:602: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:633: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:659: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:690: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:751: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:787: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:825: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:862: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:910: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:999: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1051: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1087: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1141: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1180: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_api_keys.py:1470: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/e2e/tests/test_subscription.py:1181: gateway-only mode: per-model AuthPolicy is not createdself = <test_model_identity_conflict.TestModelIdentityConflictDetection object at 0x7f97a53475b0> worker_tenant_context = WorkerTenantContext(worker_id='w4', suffix='d8a600', tenant_name='e2e-worker-w4-d8a600', tenant_namespace='ai-tenant-e...ulator-access', subscription_name='simulator-subscription', premium_subscription_name='premium-simulator-subscription') def test_colliding_model_names_flagged_then_resolved(self, worker_tenant_context): model_namespace = worker_tenant_context.model_namespace gateway_name = worker_tenant_context.gateway_name suffix = uuid.uuid4().hex[:8] shared_model_name = f"test/e2e-identity-conflict-{suffix}" llmis_a = f"e2e-conflict-a-{suffix}" llmis_b = f"e2e-conflict-b-{suffix}" ref_a, ref_b = llmis_a, llmis_b try: # A single model with no siblings must be reported unique. _create_llmis(llmis_a, model_namespace, gateway_name, GATEWAY_NAMESPACE, model_name=shared_model_name) wait_for_llmisvc_backend_ready(llmis_a, model_namespace, gateway_name) _create_maas_model_ref( ref_a, model_namespace, llmis_a, tenant_ref=worker_tenant_context.tenant_name, ) obj_a = wait_for_status_condition( "maasmodelref", ref_a, model_namespace, condition_type=CONDITION, expected_status="True", timeout=120, ) assert (obj_a.get("status") or {}).get("resolvedModelAlias"), ( f"expected {ref_a} to have a resolvedModelAlias once the LLMIS exists" ) # A second MaaSModelRef whose LLMIS declares the SAME spec.model.name # introduces a collision — this mirrors the reported production bug. _create_llmis(llmis_b, model_namespace, gateway_name, GATEWAY_NAMESPACE, model_name=shared_model_name) > wait_for_llmisvc_backend_ready(llmis_b, model_namespace, gateway_name) test/e2e/tests/test_model_identity_conflict.py:104: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'e2e-conflict-b-8b493098' namespace = 'e2e-models-e2e-worker-w4-d8a600' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/e2e-conflict-b-8b493098 in e2e-models-e2e-worker-w4-d8a600 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"e2e-conflict-b-8b493098","namespace":"e2e-models-e2e-worker-w4-d8a600"},"spec":{"model":{"name":"test/e2e-identity-conflict-8b493098","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w4-d8a600","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","test/e2e-identity-conflict-8b493098","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-16T18:40:59Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'e2e-conflict-b-8b493098', 'namespace': 'e2e-models-e2e-worker-w4-d8a600', 'resourceVersion': '43083', 'uid': '65dd69df-bfcf-4663-b5e6-fed01e6229cb'}, 'spec': {'model': {'name': 'test/e2e-identity-conflict-8b493098', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w4-d8a600', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'test/e2e-identity-conflict-8b493098', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w4-d8a600/models/test/e2e-identity-conflict-8b493098'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w4-d8a600', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w4-d8a600-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w4-d8a600/models/test/e2e-identity-conflict-8b493098'}, {'name': 'test/e2e-identity-conflict-8b493098'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w4-d8a600', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w4-d8a600-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-16T18:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-16T18:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-16T18:41:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-16T18:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-16T18:41:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-16T18:41:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-16T18:41:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'e2e-conflict-b-8b493098-kserve-route', 'namespace': 'e2e-models-e2e-worker-w4-d8a600'}], 'kind': 'Gateway', 'name': 'e2e-worker-w4-d8a600', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w4-d8a600-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'e2e-conflict-b-8b493098-kserve'}, 'service': {'kind': 'Service', 'name': 'e2e-conflict-b-8b493098-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionError/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:189: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deploy/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:355: Dormant-mode test mutates controller flags; set ENABLE_TENANT_DISCOVERY_DORMANT_E2E=true/workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners.self = <test_multi_tenant_integration.TestMultiTenantIntegration object at 0x7f5934a7d6d0> def test_full_tenant_lifecycle_create_to_delete(self): """7.1: Full tenant lifecycle from create through policy/subscription reconcile to delete.""" case = new_discovery_case() role_name = f"aitenant-{case['tenant_label_name']}-tenant-admin" try: bootstrap_aitenant_tenant(case) tenant = wait_for_json(TENANT_CONFIG_KIND, TENANT_CR_NAME, case["tenant_ns"], timeout=180) tenant_labels = tenant["metadata"].get("labels") or {} tenant_annotations = tenant["metadata"].get("annotations") or {} assert tenant_labels[LABEL_MANAGED_BY_AITENANT] == "true" assert tenant_labels[LABEL_TENANT_NAME] == case["tenant_label_name"] assert tenant_labels[LABEL_TENANT_NAMESPACE] == case["tenant_ns"] assert tenant_annotations[ANNOTATION_AITENANT_NAME] == case["tenant_label_name"] assert tenant_annotations[ANNOTATION_AITENANT_NAMESPACE] == AITENANT_NAMESPACE aitenant = wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, timeout=180) assert aitenant["status"]["gatewayRef"]["name"] == case["gateway_name"] assert get_json_or_none("role", role_name, case["tenant_ns"]) is not None model_name = f"e2e-lifecycle-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_multi_tenant_integration.py:94: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:959: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'e2e-lifecycle-model-ab83f76b' namespace = 'ai-tenant-e2e-mt-ab83f76b' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/e2e-lifecycle-model-ab83f76b in ai-tenant-e2e-mt-ab83f76b did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"e2e-lifecycle-model-ab83f76b","namespace":"ai-tenant-e2e-mt-ab83f76b"},"spec":{"model":{"name":"facebook/opt-125m","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-mt-ab83f76b","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","facebook/opt-125m","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-16T18:42:26Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'e2e-lifecycle-model-ab83f76b', 'namespace': 'ai-tenant-e2e-mt-ab83f76b', 'resourceVersion': '46871', 'uid': '4c99fe7f-4092-47cd-9cb3-96d67db18ca0'}, 'spec': {'model': {'name': 'facebook/opt-125m', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-mt-ab83f76b', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'facebook/opt-125m', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/ai-tenant-e2e-mt-ab83f76b/models/facebook/opt-125m'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-mt-ab83f76b', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-mt-ab83f76b-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/ai-tenant-e2e-mt-ab83f76b/models/facebook/opt-125m'}, {'name': 'facebook/opt-125m'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-mt-ab83f76b', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-mt-ab83f76b-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-16T18:44:22Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-16T18:44:22Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-16T18:44:22Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-16T18:44:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-16T18:44:22Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-16T18:44:22Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-16T18:44:22Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'e2e-lifecycle-model-ab83f76b-kserve-route', 'namespace': 'ai-tenant-e2e-mt-ab83f76b'}], 'kind': 'Gateway', 'name': 'e2e-mt-ab83f76b', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-mt-ab83f76b-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'e2e-lifecycle-model-ab83f76b-kserve'}, 'service': {'kind': 'Service', 'name': 'e2e-lifecycle-model-ab83f76b-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_subscription.TestStatusReporting object at 0x7fcaa1fb3340> request = <SubRequest '_worker_status_models' for <Function test_subscription_active_status_with_valid_model>> @pytest.fixture(scope="class", autouse=True) def _worker_status_models(self, request): """Provision the optional models used by status-reporting tests.""" from worker_tenant_fixtures import ensure_worker_models, serial_only_selection if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, (context.distinct_model_ref, context.distinct_model_2_ref), ) test/e2e/tests/test_subscription.py:1887: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-2-w0-909a40' namespace = 'e2e-models-e2e-worker-w0-909a40' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-2-w0-909a40 in e2e-models-e2e-worker-w0-909a40 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-2-w0-909a40","namespace":"e2e-models-e2e-worker-w0-909a40"},"spec":{"model":{"name":"e2e/distinct-2-w0-909a40","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-909a40","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-2-w0-909a40","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-16T18:41:08Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-2-w0-909a40', 'namespace': 'e2e-models-e2e-worker-w0-909a40', 'resourceVersion': '46039', 'uid': 'eb77eea0-43b1-4121-9e00-3077c9f6e3f0'}, 'spec': {'model': {'name': 'e2e/distinct-2-w0-909a40', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-2-w0-909a40', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-909a40/models/e2e/distinct-2-w0-909a40'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-909a40/models/e2e/distinct-2-w0-909a40'}, {'name': 'e2e/distinct-2-w0-909a40'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-2-w0-909a40-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-909a40'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-2-w0-909a40-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-2-w0-909a40-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_subscription.TestStatusReporting object at 0x7fcaa1fb3340> request = <SubRequest '_worker_status_models' for <Function test_subscription_active_status_with_valid_model>> @pytest.fixture(scope="class", autouse=True) def _worker_status_models(self, request): """Provision the optional models used by status-reporting tests.""" from worker_tenant_fixtures import ensure_worker_models, serial_only_selection if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, (context.distinct_model_ref, context.distinct_model_2_ref), ) test/e2e/tests/test_subscription.py:1887: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-2-w0-909a40' namespace = 'e2e-models-e2e-worker-w0-909a40' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-2-w0-909a40 in e2e-models-e2e-worker-w0-909a40 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-2-w0-909a40","namespace":"e2e-models-e2e-worker-w0-909a40"},"spec":{"model":{"name":"e2e/distinct-2-w0-909a40","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-909a40","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-2-w0-909a40","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-16T18:41:08Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-2-w0-909a40', 'namespace': 'e2e-models-e2e-worker-w0-909a40', 'resourceVersion': '46039', 'uid': 'eb77eea0-43b1-4121-9e00-3077c9f6e3f0'}, 'spec': {'model': {'name': 'e2e/distinct-2-w0-909a40', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-2-w0-909a40', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-909a40/models/e2e/distinct-2-w0-909a40'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-909a40/models/e2e/distinct-2-w0-909a40'}, {'name': 'e2e/distinct-2-w0-909a40'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-2-w0-909a40-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-909a40'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-2-w0-909a40-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-2-w0-909a40-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_subscription.TestStatusReporting object at 0x7fcaa1fb3340> request = <SubRequest '_worker_status_models' for <Function test_subscription_active_status_with_valid_model>> @pytest.fixture(scope="class", autouse=True) def _worker_status_models(self, request): """Provision the optional models used by status-reporting tests.""" from worker_tenant_fixtures import ensure_worker_models, serial_only_selection if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, (context.distinct_model_ref, context.distinct_model_2_ref), ) test/e2e/tests/test_subscription.py:1887: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-2-w0-909a40' namespace = 'e2e-models-e2e-worker-w0-909a40' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-2-w0-909a40 in e2e-models-e2e-worker-w0-909a40 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-2-w0-909a40","namespace":"e2e-models-e2e-worker-w0-909a40"},"spec":{"model":{"name":"e2e/distinct-2-w0-909a40","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-909a40","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-2-w0-909a40","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-16T18:41:08Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-2-w0-909a40', 'namespace': 'e2e-models-e2e-worker-w0-909a40', 'resourceVersion': '46039', 'uid': 'eb77eea0-43b1-4121-9e00-3077c9f6e3f0'}, 'spec': {'model': {'name': 'e2e/distinct-2-w0-909a40', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-2-w0-909a40', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-909a40/models/e2e/distinct-2-w0-909a40'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-909a40/models/e2e/distinct-2-w0-909a40'}, {'name': 'e2e/distinct-2-w0-909a40'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-2-w0-909a40-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-909a40'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-2-w0-909a40-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-2-w0-909a40-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_subscription.TestStatusReporting object at 0x7fcaa1fb3340> request = <SubRequest '_worker_status_models' for <Function test_subscription_active_status_with_valid_model>> @pytest.fixture(scope="class", autouse=True) def _worker_status_models(self, request): """Provision the optional models used by status-reporting tests.""" from worker_tenant_fixtures import ensure_worker_models, serial_only_selection if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, (context.distinct_model_ref, context.distinct_model_2_ref), ) test/e2e/tests/test_subscription.py:1887: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-2-w0-909a40' namespace = 'e2e-models-e2e-worker-w0-909a40' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-2-w0-909a40 in e2e-models-e2e-worker-w0-909a40 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-2-w0-909a40","namespace":"e2e-models-e2e-worker-w0-909a40"},"spec":{"model":{"name":"e2e/distinct-2-w0-909a40","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-909a40","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-2-w0-909a40","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-16T18:41:08Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-2-w0-909a40', 'namespace': 'e2e-models-e2e-worker-w0-909a40', 'resourceVersion': '46039', 'uid': 'eb77eea0-43b1-4121-9e00-3077c9f6e3f0'}, 'spec': {'model': {'name': 'e2e/distinct-2-w0-909a40', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-2-w0-909a40', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-909a40/models/e2e/distinct-2-w0-909a40'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-909a40/models/e2e/distinct-2-w0-909a40'}, {'name': 'e2e/distinct-2-w0-909a40'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-2-w0-909a40-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-909a40'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-2-w0-909a40-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-2-w0-909a40-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_subscription.TestStatusReporting object at 0x7fcaa1fb3340> request = <SubRequest '_worker_status_models' for <Function test_subscription_active_status_with_valid_model>> @pytest.fixture(scope="class", autouse=True) def _worker_status_models(self, request): """Provision the optional models used by status-reporting tests.""" from worker_tenant_fixtures import ensure_worker_models, serial_only_selection if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, (context.distinct_model_ref, context.distinct_model_2_ref), ) test/e2e/tests/test_subscription.py:1887: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-2-w0-909a40' namespace = 'e2e-models-e2e-worker-w0-909a40' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-2-w0-909a40 in e2e-models-e2e-worker-w0-909a40 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-2-w0-909a40","namespace":"e2e-models-e2e-worker-w0-909a40"},"spec":{"model":{"name":"e2e/distinct-2-w0-909a40","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-909a40","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-2-w0-909a40","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-16T18:41:08Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-2-w0-909a40', 'namespace': 'e2e-models-e2e-worker-w0-909a40', 'resourceVersion': '46039', 'uid': 'eb77eea0-43b1-4121-9e00-3077c9f6e3f0'}, 'spec': {'model': {'name': 'e2e/distinct-2-w0-909a40', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-2-w0-909a40', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-909a40/models/e2e/distinct-2-w0-909a40'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-909a40/models/e2e/distinct-2-w0-909a40'}, {'name': 'e2e/distinct-2-w0-909a40'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-16T18:43:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-2-w0-909a40-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-909a40'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-909a40', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-909a40-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-2-w0-909a40-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-2-w0-909a40-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorshared_test_tenants = ({'base_url': 'https://e2e-shared-a-w3-8edc30.apps.7b789c0d-5c81-456e-8eda-5095411539b7.prod.konfluxeaas.com/maas-api'...me': 'e2e-shared-b-w3-ee3af0', 'name': 'e2e-shared-b-w3-ee3af0', 'namespace': 'ai-tenant-e2e-shared-b-w3-ee3af0', ...}) @pytest.fixture(scope="module") def tenant_env(shared_test_tenants): """Adapter fixture with tenant-specific models.""" case_a, case_b = dict(shared_test_tenants[0]), dict(shared_test_tenants[1]) for case in (case_a, case_b): model_name = f"auth-test-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_tenant_auth_isolation.py:99: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:959: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'auth-test-model-8edc30' namespace = 'ai-tenant-e2e-shared-a-w3-8edc30' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/auth-test-model-8edc30 in ai-tenant-e2e-shared-a-w3-8edc30 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"auth-test-model-8edc30","namespace":"ai-tenant-e2e-shared-a-w3-8edc30"},"spec":{"model":{"name":"facebook/opt-125m","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-shared-a-w3-8edc30","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","facebook/opt-125m","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-16T18:42:36Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'auth-test-model-8edc30', 'namespace': 'ai-tenant-e2e-shared-a-w3-8edc30', 'resourceVersion': '47544', 'uid': '079ea753-d463-43d3-8133-0282c1ed7e8a'}, 'spec': {'model': {'name': 'facebook/opt-125m', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'facebook/opt-125m', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/ai-tenant-e2e-shared-a-w3-8edc30/models/facebook/opt-125m'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/ai-tenant-e2e-shared-a-w3-8edc30/models/facebook/opt-125m'}, {'name': 'facebook/opt-125m'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'auth-test-model-8edc30-kserve-route', 'namespace': 'ai-tenant-e2e-shared-a-w3-8edc30'}], 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'auth-test-model-8edc30-kserve'}, 'service': {'kind': 'Service', 'name': 'auth-test-model-8edc30-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorshared_test_tenants = ({'base_url': 'https://e2e-shared-a-w3-8edc30.apps.7b789c0d-5c81-456e-8eda-5095411539b7.prod.konfluxeaas.com/maas-api'...me': 'e2e-shared-b-w3-ee3af0', 'name': 'e2e-shared-b-w3-ee3af0', 'namespace': 'ai-tenant-e2e-shared-b-w3-ee3af0', ...}) @pytest.fixture(scope="module") def tenant_env(shared_test_tenants): """Adapter fixture with tenant-specific models.""" case_a, case_b = dict(shared_test_tenants[0]), dict(shared_test_tenants[1]) for case in (case_a, case_b): model_name = f"auth-test-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_tenant_auth_isolation.py:99: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:959: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'auth-test-model-8edc30' namespace = 'ai-tenant-e2e-shared-a-w3-8edc30' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/auth-test-model-8edc30 in ai-tenant-e2e-shared-a-w3-8edc30 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"auth-test-model-8edc30","namespace":"ai-tenant-e2e-shared-a-w3-8edc30"},"spec":{"model":{"name":"facebook/opt-125m","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-shared-a-w3-8edc30","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","facebook/opt-125m","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-16T18:42:36Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'auth-test-model-8edc30', 'namespace': 'ai-tenant-e2e-shared-a-w3-8edc30', 'resourceVersion': '47544', 'uid': '079ea753-d463-43d3-8133-0282c1ed7e8a'}, 'spec': {'model': {'name': 'facebook/opt-125m', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'facebook/opt-125m', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/ai-tenant-e2e-shared-a-w3-8edc30/models/facebook/opt-125m'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/ai-tenant-e2e-shared-a-w3-8edc30/models/facebook/opt-125m'}, {'name': 'facebook/opt-125m'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'auth-test-model-8edc30-kserve-route', 'namespace': 'ai-tenant-e2e-shared-a-w3-8edc30'}], 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'auth-test-model-8edc30-kserve'}, 'service': {'kind': 'Service', 'name': 'auth-test-model-8edc30-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorshared_test_tenants = ({'base_url': 'https://e2e-shared-a-w3-8edc30.apps.7b789c0d-5c81-456e-8eda-5095411539b7.prod.konfluxeaas.com/maas-api'...me': 'e2e-shared-b-w3-ee3af0', 'name': 'e2e-shared-b-w3-ee3af0', 'namespace': 'ai-tenant-e2e-shared-b-w3-ee3af0', ...}) @pytest.fixture(scope="module") def tenant_env(shared_test_tenants): """Adapter fixture with tenant-specific models.""" case_a, case_b = dict(shared_test_tenants[0]), dict(shared_test_tenants[1]) for case in (case_a, case_b): model_name = f"auth-test-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_tenant_auth_isolation.py:99: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:959: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'auth-test-model-8edc30' namespace = 'ai-tenant-e2e-shared-a-w3-8edc30' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/auth-test-model-8edc30 in ai-tenant-e2e-shared-a-w3-8edc30 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"auth-test-model-8edc30","namespace":"ai-tenant-e2e-shared-a-w3-8edc30"},"spec":{"model":{"name":"facebook/opt-125m","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-shared-a-w3-8edc30","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","facebook/opt-125m","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-16T18:42:36Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'auth-test-model-8edc30', 'namespace': 'ai-tenant-e2e-shared-a-w3-8edc30', 'resourceVersion': '47544', 'uid': '079ea753-d463-43d3-8133-0282c1ed7e8a'}, 'spec': {'model': {'name': 'facebook/opt-125m', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'facebook/opt-125m', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/ai-tenant-e2e-shared-a-w3-8edc30/models/facebook/opt-125m'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/ai-tenant-e2e-shared-a-w3-8edc30/models/facebook/opt-125m'}, {'name': 'facebook/opt-125m'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'auth-test-model-8edc30-kserve-route', 'namespace': 'ai-tenant-e2e-shared-a-w3-8edc30'}], 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'auth-test-model-8edc30-kserve'}, 'service': {'kind': 'Service', 'name': 'auth-test-model-8edc30-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorshared_test_tenants = ({'base_url': 'https://e2e-shared-a-w3-8edc30.apps.7b789c0d-5c81-456e-8eda-5095411539b7.prod.konfluxeaas.com/maas-api'...me': 'e2e-shared-b-w3-ee3af0', 'name': 'e2e-shared-b-w3-ee3af0', 'namespace': 'ai-tenant-e2e-shared-b-w3-ee3af0', ...}) @pytest.fixture(scope="module") def tenant_env(shared_test_tenants): """Adapter fixture with tenant-specific models.""" case_a, case_b = dict(shared_test_tenants[0]), dict(shared_test_tenants[1]) for case in (case_a, case_b): model_name = f"auth-test-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_tenant_auth_isolation.py:99: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:959: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'auth-test-model-8edc30' namespace = 'ai-tenant-e2e-shared-a-w3-8edc30' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/auth-test-model-8edc30 in ai-tenant-e2e-shared-a-w3-8edc30 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"auth-test-model-8edc30","namespace":"ai-tenant-e2e-shared-a-w3-8edc30"},"spec":{"model":{"name":"facebook/opt-125m","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-shared-a-w3-8edc30","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","facebook/opt-125m","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-16T18:42:36Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'auth-test-model-8edc30', 'namespace': 'ai-tenant-e2e-shared-a-w3-8edc30', 'resourceVersion': '47544', 'uid': '079ea753-d463-43d3-8133-0282c1ed7e8a'}, 'spec': {'model': {'name': 'facebook/opt-125m', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'facebook/opt-125m', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/ai-tenant-e2e-shared-a-w3-8edc30/models/facebook/opt-125m'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/ai-tenant-e2e-shared-a-w3-8edc30/models/facebook/opt-125m'}, {'name': 'facebook/opt-125m'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'auth-test-model-8edc30-kserve-route', 'namespace': 'ai-tenant-e2e-shared-a-w3-8edc30'}], 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'auth-test-model-8edc30-kserve'}, 'service': {'kind': 'Service', 'name': 'auth-test-model-8edc30-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorshared_test_tenants = ({'base_url': 'https://e2e-shared-a-w3-8edc30.apps.7b789c0d-5c81-456e-8eda-5095411539b7.prod.konfluxeaas.com/maas-api'...me': 'e2e-shared-b-w3-ee3af0', 'name': 'e2e-shared-b-w3-ee3af0', 'namespace': 'ai-tenant-e2e-shared-b-w3-ee3af0', ...}) @pytest.fixture(scope="module") def tenant_env(shared_test_tenants): """Adapter fixture with tenant-specific models.""" case_a, case_b = dict(shared_test_tenants[0]), dict(shared_test_tenants[1]) for case in (case_a, case_b): model_name = f"auth-test-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_tenant_auth_isolation.py:99: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:959: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'auth-test-model-8edc30' namespace = 'ai-tenant-e2e-shared-a-w3-8edc30' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/auth-test-model-8edc30 in ai-tenant-e2e-shared-a-w3-8edc30 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"auth-test-model-8edc30","namespace":"ai-tenant-e2e-shared-a-w3-8edc30"},"spec":{"model":{"name":"facebook/opt-125m","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-shared-a-w3-8edc30","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","facebook/opt-125m","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-16T18:42:36Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'auth-test-model-8edc30', 'namespace': 'ai-tenant-e2e-shared-a-w3-8edc30', 'resourceVersion': '47544', 'uid': '079ea753-d463-43d3-8133-0282c1ed7e8a'}, 'spec': {'model': {'name': 'facebook/opt-125m', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'facebook/opt-125m', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/ai-tenant-e2e-shared-a-w3-8edc30/models/facebook/opt-125m'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/ai-tenant-e2e-shared-a-w3-8edc30/models/facebook/opt-125m'}, {'name': 'facebook/opt-125m'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-16T18:44:43Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'auth-test-model-8edc30-kserve-route', 'namespace': 'ai-tenant-e2e-shared-a-w3-8edc30'}], 'kind': 'Gateway', 'name': 'e2e-shared-a-w3-8edc30', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-shared-a-w3-8edc30-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'auth-test-model-8edc30-kserve'}, 'service': {'kind': 'Service', 'name': 'auth-test-model-8edc30-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionError/workspace/source/test/e2e/tests/test_gateway_scoped_authpolicy.py:219: legacy gateway-default-auth is scoped to the shared default gateway