/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_external_oidc.py:252: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:263: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:277: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:290: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:303: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:335: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:379: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:395: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:405: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:428: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:436: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:441: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:454: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:501: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:537: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:602: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:633: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:659: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:690: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:751: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:787: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:825: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:862: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:910: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:999: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1051: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1087: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1141: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1180: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:189: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deploy/workspace/source/test/e2e/tests/test_api_keys.py:1470: Cannot find maas-api pod in odh-ai-gateway-infra: error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template: template was: {.items[0].metadata.name} object given to jsonpath engine was: map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":""}}/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:355: Dormant-mode test mutates controller flags; set ENABLE_TENANT_DISCOVERY_DORMANT_E2E=true/workspace/source/test/e2e/tests/test_subscription.py:1181: gateway-only mode: per-model AuthPolicy is not created/workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners.self = <test_multi_tenant_integration.TestMultiTenantIntegration object at 0x7f1321ac5af0> def test_full_tenant_lifecycle_create_to_delete(self): """7.1: Full tenant lifecycle from create through policy/subscription reconcile to delete.""" case = new_discovery_case() role_name = f"aitenant-{case['tenant_label_name']}-tenant-admin" try: bootstrap_aitenant_tenant(case) tenant = wait_for_json(TENANT_CONFIG_KIND, TENANT_CR_NAME, case["tenant_ns"], timeout=180) tenant_labels = tenant["metadata"].get("labels") or {} tenant_annotations = tenant["metadata"].get("annotations") or {} assert tenant_labels[LABEL_MANAGED_BY_AITENANT] == "true" assert tenant_labels[LABEL_TENANT_NAME] == case["tenant_label_name"] assert tenant_labels[LABEL_TENANT_NAMESPACE] == case["tenant_ns"] assert tenant_annotations[ANNOTATION_AITENANT_NAME] == case["tenant_label_name"] assert tenant_annotations[ANNOTATION_AITENANT_NAMESPACE] == AITENANT_NAMESPACE aitenant = wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, timeout=180) assert aitenant["status"]["gatewayRef"]["name"] == case["gateway_name"] assert get_json_or_none("role", role_name, case["tenant_ns"]) is not None model_name = f"e2e-lifecycle-model-{case['suffix']}" > provision_tenant_model(model_name, case["tenant_ns"], case["gateway_name"]) test/e2e/tests/test_multi_tenant_integration.py:94: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/multitenancy_helpers.py:959: in provision_tenant_model wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'e2e-lifecycle-model-f7e72d9f' namespace = 'ai-tenant-e2e-mt-f7e72d9f' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/e2e-lifecycle-model-f7e72d9f in ai-tenant-e2e-mt-f7e72d9f did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"e2e-lifecycle-model-f7e72d9f","namespace":"ai-tenant-e2e-mt-f7e72d9f"},"spec":{"model":{"name":"facebook/opt-125m","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-mt-f7e72d9f","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","facebook/opt-125m","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-18T18:11:40Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'e2e-lifecycle-model-f7e72d9f', 'namespace': 'ai-tenant-e2e-mt-f7e72d9f', 'resourceVersion': '43127', 'uid': 'df82a6da-9662-4e34-862a-88de2bca83f8'}, 'spec': {'model': {'name': 'facebook/opt-125m', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-mt-f7e72d9f', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'facebook/opt-125m', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/ai-tenant-e2e-mt-f7e72d9f/models/facebook/opt-125m'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-mt-f7e72d9f', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-mt-f7e72d9f-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/ai-tenant-e2e-mt-f7e72d9f/models/facebook/opt-125m'}, {'name': 'facebook/opt-125m'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-mt-f7e72d9f', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-mt-f7e72d9f-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-mt-f7e72d9f/e2e-lifecycle-model-f7e72d9f'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-18T18:13:23Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-18T18:13:23Z', 'message': 'The following HTTPRoutes are not ready: [ai-tenant-e2e-mt-f7e72d9f/e2e-lifecycle-model-f7e72d9f-kserve-route: v1.HTTPRouteStatus{RouteStatus:v1.RouteStatus{Parents:[]v1.RouteParentStatus(nil)}}]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T18:13:23Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T18:13:23Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T18:13:23Z', 'message': 'The following HTTPRoutes are not ready: [ai-tenant-e2e-mt-f7e72d9f/e2e-lifecycle-model-f7e72d9f-kserve-route: v1.HTTPRouteStatus{RouteStatus:v1.RouteStatus{Parents:[]v1.RouteParentStatus(nil)}}]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T18:13:23Z', 'message': 'The following HTTPRoutes are not ready: [ai-tenant-e2e-mt-f7e72d9f/e2e-lifecycle-model-f7e72d9f-kserve-route: v1.HTTPRouteStatus{RouteStatus:v1.RouteStatus{Parents:[]v1.RouteParentStatus(nil)}}]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T18:13:23Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'e2e-lifecycle-model-f7e72d9f-kserve-route', 'namespace': 'ai-tenant-e2e-mt-f7e72d9f'}], 'kind': 'Gateway', 'name': 'e2e-mt-f7e72d9f', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-mt-f7e72d9f-openshift-default.openshift-ingress.svc.cluster.local/ai-tenant-e2e-mt-f7e72d9f/e2e-lifecycle-model-f7e72d9f', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'e2e-lifecycle-model-f7e72d9f-kserve'}, 'service': {'kind': 'Service', 'name': 'e2e-lifecycle-model-f7e72d9f-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_subscription.TestStatusReporting object at 0x7efc34692c40> request = <SubRequest '_worker_status_models' for <Function test_subscription_active_status_with_valid_model>> @pytest.fixture(scope="class", autouse=True) def _worker_status_models(self, request): """Provision the optional models used by status-reporting tests.""" from worker_tenant_fixtures import ensure_worker_models, serial_only_selection if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, (context.distinct_model_ref, context.distinct_model_2_ref), ) test/e2e/tests/test_subscription.py:1887: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w0-609aba' namespace = 'e2e-models-e2e-worker-w0-609aba' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-w0-609aba in e2e-models-e2e-worker-w0-609aba did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-w0-609aba","namespace":"e2e-models-e2e-worker-w0-609aba"},"spec":{"model":{"name":"e2e/distinct-w0-609aba","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-609aba","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-w0-609aba","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-18T18:10:15Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-w0-609aba', 'namespace': 'e2e-models-e2e-worker-w0-609aba', 'resourceVersion': '41157', 'uid': '00a2d9cb-78ff-4952-af83-d967931693aa'}, 'spec': {'model': {'name': 'e2e/distinct-w0-609aba', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-w0-609aba', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-609aba/models/e2e/distinct-w0-609aba'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-609aba/models/e2e/distinct-w0-609aba'}, {'name': 'e2e/distinct-w0-609aba'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-609aba/distinct-w0-609aba'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-w0-609aba-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-609aba'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-609aba/distinct-w0-609aba', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-w0-609aba-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-w0-609aba-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_subscription.TestStatusReporting object at 0x7efc34692c40> request = <SubRequest '_worker_status_models' for <Function test_subscription_active_status_with_valid_model>> @pytest.fixture(scope="class", autouse=True) def _worker_status_models(self, request): """Provision the optional models used by status-reporting tests.""" from worker_tenant_fixtures import ensure_worker_models, serial_only_selection if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, (context.distinct_model_ref, context.distinct_model_2_ref), ) test/e2e/tests/test_subscription.py:1887: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w0-609aba' namespace = 'e2e-models-e2e-worker-w0-609aba' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-w0-609aba in e2e-models-e2e-worker-w0-609aba did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-w0-609aba","namespace":"e2e-models-e2e-worker-w0-609aba"},"spec":{"model":{"name":"e2e/distinct-w0-609aba","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-609aba","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-w0-609aba","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-18T18:10:15Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-w0-609aba', 'namespace': 'e2e-models-e2e-worker-w0-609aba', 'resourceVersion': '41157', 'uid': '00a2d9cb-78ff-4952-af83-d967931693aa'}, 'spec': {'model': {'name': 'e2e/distinct-w0-609aba', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-w0-609aba', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-609aba/models/e2e/distinct-w0-609aba'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-609aba/models/e2e/distinct-w0-609aba'}, {'name': 'e2e/distinct-w0-609aba'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-609aba/distinct-w0-609aba'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-w0-609aba-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-609aba'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-609aba/distinct-w0-609aba', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-w0-609aba-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-w0-609aba-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_subscription.TestStatusReporting object at 0x7efc34692c40> request = <SubRequest '_worker_status_models' for <Function test_subscription_active_status_with_valid_model>> @pytest.fixture(scope="class", autouse=True) def _worker_status_models(self, request): """Provision the optional models used by status-reporting tests.""" from worker_tenant_fixtures import ensure_worker_models, serial_only_selection if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, (context.distinct_model_ref, context.distinct_model_2_ref), ) test/e2e/tests/test_subscription.py:1887: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w0-609aba' namespace = 'e2e-models-e2e-worker-w0-609aba' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-w0-609aba in e2e-models-e2e-worker-w0-609aba did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-w0-609aba","namespace":"e2e-models-e2e-worker-w0-609aba"},"spec":{"model":{"name":"e2e/distinct-w0-609aba","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-609aba","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-w0-609aba","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-18T18:10:15Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-w0-609aba', 'namespace': 'e2e-models-e2e-worker-w0-609aba', 'resourceVersion': '41157', 'uid': '00a2d9cb-78ff-4952-af83-d967931693aa'}, 'spec': {'model': {'name': 'e2e/distinct-w0-609aba', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-w0-609aba', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-609aba/models/e2e/distinct-w0-609aba'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-609aba/models/e2e/distinct-w0-609aba'}, {'name': 'e2e/distinct-w0-609aba'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-609aba/distinct-w0-609aba'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-w0-609aba-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-609aba'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-609aba/distinct-w0-609aba', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-w0-609aba-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-w0-609aba-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_subscription.TestStatusReporting object at 0x7efc34692c40> request = <SubRequest '_worker_status_models' for <Function test_subscription_active_status_with_valid_model>> @pytest.fixture(scope="class", autouse=True) def _worker_status_models(self, request): """Provision the optional models used by status-reporting tests.""" from worker_tenant_fixtures import ensure_worker_models, serial_only_selection if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, (context.distinct_model_ref, context.distinct_model_2_ref), ) test/e2e/tests/test_subscription.py:1887: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w0-609aba' namespace = 'e2e-models-e2e-worker-w0-609aba' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-w0-609aba in e2e-models-e2e-worker-w0-609aba did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-w0-609aba","namespace":"e2e-models-e2e-worker-w0-609aba"},"spec":{"model":{"name":"e2e/distinct-w0-609aba","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-609aba","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-w0-609aba","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-18T18:10:15Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-w0-609aba', 'namespace': 'e2e-models-e2e-worker-w0-609aba', 'resourceVersion': '41157', 'uid': '00a2d9cb-78ff-4952-af83-d967931693aa'}, 'spec': {'model': {'name': 'e2e/distinct-w0-609aba', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-w0-609aba', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-609aba/models/e2e/distinct-w0-609aba'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-609aba/models/e2e/distinct-w0-609aba'}, {'name': 'e2e/distinct-w0-609aba'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-609aba/distinct-w0-609aba'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-w0-609aba-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-609aba'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-609aba/distinct-w0-609aba', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-w0-609aba-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-w0-609aba-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorself = <test_subscription.TestStatusReporting object at 0x7efc34692c40> request = <SubRequest '_worker_status_models' for <Function test_subscription_active_status_with_valid_model>> @pytest.fixture(scope="class", autouse=True) def _worker_status_models(self, request): """Provision the optional models used by status-reporting tests.""" from worker_tenant_fixtures import ensure_worker_models, serial_only_selection if serial_only_selection(request): yield return context = request.getfixturevalue("worker_tenant_context") > ensure_worker_models( context, (context.distinct_model_ref, context.distinct_model_2_ref), ) test/e2e/tests/test_subscription.py:1887: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:304: in ensure_worker_models wait_for_llmisvc_backend_ready( test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'distinct-w0-609aba' namespace = 'e2e-models-e2e-worker-w0-609aba' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/distinct-w0-609aba in e2e-models-e2e-worker-w0-609aba did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"distinct-w0-609aba","namespace":"e2e-models-e2e-worker-w0-609aba"},"spec":{"model":{"name":"e2e/distinct-w0-609aba","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-609aba","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/distinct-w0-609aba","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-18T18:10:15Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'distinct-w0-609aba', 'namespace': 'e2e-models-e2e-worker-w0-609aba', 'resourceVersion': '41157', 'uid': '00a2d9cb-78ff-4952-af83-d967931693aa'}, 'spec': {'model': {'name': 'e2e/distinct-w0-609aba', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/distinct-w0-609aba', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-609aba/models/e2e/distinct-w0-609aba'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-609aba/models/e2e/distinct-w0-609aba'}, {'name': 'e2e/distinct-w0-609aba'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-609aba/distinct-w0-609aba'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T18:11:46Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'distinct-w0-609aba-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-609aba'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-609aba', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-609aba-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-609aba/distinct-w0-609aba', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'distinct-w0-609aba-kserve'}, 'service': {'kind': 'Service', 'name': 'distinct-w0-609aba-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionError/workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:215: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B/workspace/source/test/e2e/tests/test_gateway_scoped_authpolicy.py:219: legacy gateway-default-auth is scoped to the shared default gateway