/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_external_oidc.py:252: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:263: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:277: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:290: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:303: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:335: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:379: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:395: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:405: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:428: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:436: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:441: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:454: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:501: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:537: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:602: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:633: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:659: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:690: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:750: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:786: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:824: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:872: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:961: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1013: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1049: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1103: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1142: EXTERNAL_OIDC not enabled@pytest.fixture(scope="session") def worker_tenant_context(): """Create one isolated AITenant for tests explicitly marked worker_tenant.""" from worker_tenant_fixtures import ( bootstrap_worker_tenant, build_worker_tenant_case, teardown_worker_tenant, worker_tenant_enabled, ) if not worker_tenant_enabled(): pytest.skip("worker tenant isolation disabled by E2E_USE_WORKER_TENANT") case = build_worker_tenant_case(_xdist_worker_suffix()) try: > case = bootstrap_worker_tenant(case) test/e2e/tests/conftest.py:37: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:253: in bootstrap_worker_tenant wait_for_llmisvc_backend_ready(model_ref, context.model_namespace, context.gateway_name) test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'simulator-w4-962a7d' namespace = 'e2e-models-e2e-worker-w4-962a7d' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/simulator-w4-962a7d in e2e-models-e2e-worker-w4-962a7d did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"simulator-w4-962a7d","namespace":"e2e-models-e2e-worker-w4-962a7d"},"spec":{"model":{"name":"e2e/simulator-w4-962a7d","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w4-962a7d","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/simulator-w4-962a7d","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T21:17:12Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'simulator-w4-962a7d', 'namespace': 'e2e-models-e2e-worker-w4-962a7d', 'resourceVersion': '29865', 'uid': '0564e9b0-8fcc-4b37-b67d-9d9c3ac820a8'}, 'spec': {'model': {'name': 'e2e/simulator-w4-962a7d', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/simulator-w4-962a7d', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w4-962a7d/models/e2e/simulator-w4-962a7d'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w4-962a7d/models/e2e/simulator-w4-962a7d'}, {'name': 'e2e/simulator-w4-962a7d'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'simulator-w4-962a7d-kserve-route', 'namespace': 'e2e-models-e2e-worker-w4-962a7d'}], 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'simulator-w4-962a7d-kserve'}, 'service': {'kind': 'Service', 'name': 'simulator-w4-962a7d-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_models_endpoint.py:92: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1108: in execute raise exc.with_traceback(exc_tb) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:253: in bootstrap_worker_tenant wait_for_llmisvc_backend_ready(model_ref, context.model_namespace, context.gateway_name) test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'simulator-w4-962a7d' namespace = 'e2e-models-e2e-worker-w4-962a7d' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/simulator-w4-962a7d in e2e-models-e2e-worker-w4-962a7d did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"simulator-w4-962a7d","namespace":"e2e-models-e2e-worker-w4-962a7d"},"spec":{"model":{"name":"e2e/simulator-w4-962a7d","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w4-962a7d","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/simulator-w4-962a7d","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T21:17:12Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'simulator-w4-962a7d', 'namespace': 'e2e-models-e2e-worker-w4-962a7d', 'resourceVersion': '29865', 'uid': '0564e9b0-8fcc-4b37-b67d-9d9c3ac820a8'}, 'spec': {'model': {'name': 'e2e/simulator-w4-962a7d', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/simulator-w4-962a7d', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w4-962a7d/models/e2e/simulator-w4-962a7d'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w4-962a7d/models/e2e/simulator-w4-962a7d'}, {'name': 'e2e/simulator-w4-962a7d'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'simulator-w4-962a7d-kserve-route', 'namespace': 'e2e-models-e2e-worker-w4-962a7d'}], 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'simulator-w4-962a7d-kserve'}, 'service': {'kind': 'Service', 'name': 'simulator-w4-962a7d-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_models_endpoint.py:92: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1108: in execute raise exc.with_traceback(exc_tb) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:253: in bootstrap_worker_tenant wait_for_llmisvc_backend_ready(model_ref, context.model_namespace, context.gateway_name) test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'simulator-w4-962a7d' namespace = 'e2e-models-e2e-worker-w4-962a7d' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/simulator-w4-962a7d in e2e-models-e2e-worker-w4-962a7d did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"simulator-w4-962a7d","namespace":"e2e-models-e2e-worker-w4-962a7d"},"spec":{"model":{"name":"e2e/simulator-w4-962a7d","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w4-962a7d","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/simulator-w4-962a7d","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T21:17:12Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'simulator-w4-962a7d', 'namespace': 'e2e-models-e2e-worker-w4-962a7d', 'resourceVersion': '29865', 'uid': '0564e9b0-8fcc-4b37-b67d-9d9c3ac820a8'}, 'spec': {'model': {'name': 'e2e/simulator-w4-962a7d', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/simulator-w4-962a7d', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w4-962a7d/models/e2e/simulator-w4-962a7d'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w4-962a7d/models/e2e/simulator-w4-962a7d'}, {'name': 'e2e/simulator-w4-962a7d'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'simulator-w4-962a7d-kserve-route', 'namespace': 'e2e-models-e2e-worker-w4-962a7d'}], 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'simulator-w4-962a7d-kserve'}, 'service': {'kind': 'Service', 'name': 'simulator-w4-962a7d-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_models_endpoint.py:92: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1108: in execute raise exc.with_traceback(exc_tb) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:253: in bootstrap_worker_tenant wait_for_llmisvc_backend_ready(model_ref, context.model_namespace, context.gateway_name) test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'simulator-w4-962a7d' namespace = 'e2e-models-e2e-worker-w4-962a7d' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/simulator-w4-962a7d in e2e-models-e2e-worker-w4-962a7d did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"simulator-w4-962a7d","namespace":"e2e-models-e2e-worker-w4-962a7d"},"spec":{"model":{"name":"e2e/simulator-w4-962a7d","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w4-962a7d","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/simulator-w4-962a7d","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T21:17:12Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'simulator-w4-962a7d', 'namespace': 'e2e-models-e2e-worker-w4-962a7d', 'resourceVersion': '29865', 'uid': '0564e9b0-8fcc-4b37-b67d-9d9c3ac820a8'}, 'spec': {'model': {'name': 'e2e/simulator-w4-962a7d', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/simulator-w4-962a7d', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w4-962a7d/models/e2e/simulator-w4-962a7d'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w4-962a7d/models/e2e/simulator-w4-962a7d'}, {'name': 'e2e/simulator-w4-962a7d'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'simulator-w4-962a7d-kserve-route', 'namespace': 'e2e-models-e2e-worker-w4-962a7d'}], 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'simulator-w4-962a7d-kserve'}, 'service': {'kind': 'Service', 'name': 'simulator-w4-962a7d-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_models_endpoint.py:92: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1108: in execute raise exc.with_traceback(exc_tb) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:253: in bootstrap_worker_tenant wait_for_llmisvc_backend_ready(model_ref, context.model_namespace, context.gateway_name) test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'simulator-w4-962a7d' namespace = 'e2e-models-e2e-worker-w4-962a7d' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/simulator-w4-962a7d in e2e-models-e2e-worker-w4-962a7d did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"simulator-w4-962a7d","namespace":"e2e-models-e2e-worker-w4-962a7d"},"spec":{"model":{"name":"e2e/simulator-w4-962a7d","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w4-962a7d","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/simulator-w4-962a7d","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T21:17:12Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'simulator-w4-962a7d', 'namespace': 'e2e-models-e2e-worker-w4-962a7d', 'resourceVersion': '29865', 'uid': '0564e9b0-8fcc-4b37-b67d-9d9c3ac820a8'}, 'spec': {'model': {'name': 'e2e/simulator-w4-962a7d', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/simulator-w4-962a7d', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w4-962a7d/models/e2e/simulator-w4-962a7d'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w4-962a7d/models/e2e/simulator-w4-962a7d'}, {'name': 'e2e/simulator-w4-962a7d'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T21:18:42Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'simulator-w4-962a7d-kserve-route', 'namespace': 'e2e-models-e2e-worker-w4-962a7d'}], 'kind': 'Gateway', 'name': 'e2e-worker-w4-962a7d', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w4-962a7d-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'simulator-w4-962a7d-kserve'}, 'service': {'kind': 'Service', 'name': 'simulator-w4-962a7d-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_api_keys_context' for <Function test_create_api_key>> @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:253: in bootstrap_worker_tenant wait_for_llmisvc_backend_ready(model_ref, context.model_namespace, context.gateway_name) test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'simulator-w0-ea5e33' namespace = 'e2e-models-e2e-worker-w0-ea5e33' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/simulator-w0-ea5e33 in e2e-models-e2e-worker-w0-ea5e33 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"simulator-w0-ea5e33","namespace":"e2e-models-e2e-worker-w0-ea5e33"},"spec":{"model":{"name":"e2e/simulator-w0-ea5e33","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-ea5e33","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/simulator-w0-ea5e33","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T21:17:12Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'simulator-w0-ea5e33', 'namespace': 'e2e-models-e2e-worker-w0-ea5e33', 'resourceVersion': '28372', 'uid': 'f4f0a941-6134-4b27-b5e7-fbb9080e60cf'}, 'spec': {'model': {'name': 'e2e/simulator-w0-ea5e33', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/simulator-w0-ea5e33', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-ea5e33/models/e2e/simulator-w0-ea5e33'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-ea5e33/models/e2e/simulator-w0-ea5e33'}, {'name': 'e2e/simulator-w0-ea5e33'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'simulator-w0-ea5e33-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-ea5e33'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'simulator-w0-ea5e33-kserve'}, 'service': {'kind': 'Service', 'name': 'simulator-w0-ea5e33-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_api_keys_context' for <Function test_create_api_key>> @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:253: in bootstrap_worker_tenant wait_for_llmisvc_backend_ready(model_ref, context.model_namespace, context.gateway_name) test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'simulator-w0-ea5e33' namespace = 'e2e-models-e2e-worker-w0-ea5e33' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/simulator-w0-ea5e33 in e2e-models-e2e-worker-w0-ea5e33 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"simulator-w0-ea5e33","namespace":"e2e-models-e2e-worker-w0-ea5e33"},"spec":{"model":{"name":"e2e/simulator-w0-ea5e33","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-ea5e33","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/simulator-w0-ea5e33","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T21:17:12Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'simulator-w0-ea5e33', 'namespace': 'e2e-models-e2e-worker-w0-ea5e33', 'resourceVersion': '28372', 'uid': 'f4f0a941-6134-4b27-b5e7-fbb9080e60cf'}, 'spec': {'model': {'name': 'e2e/simulator-w0-ea5e33', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/simulator-w0-ea5e33', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-ea5e33/models/e2e/simulator-w0-ea5e33'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-ea5e33/models/e2e/simulator-w0-ea5e33'}, {'name': 'e2e/simulator-w0-ea5e33'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'simulator-w0-ea5e33-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-ea5e33'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'simulator-w0-ea5e33-kserve'}, 'service': {'kind': 'Service', 'name': 'simulator-w0-ea5e33-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_api_keys_context' for <Function test_create_api_key>> @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:253: in bootstrap_worker_tenant wait_for_llmisvc_backend_ready(model_ref, context.model_namespace, context.gateway_name) test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'simulator-w0-ea5e33' namespace = 'e2e-models-e2e-worker-w0-ea5e33' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/simulator-w0-ea5e33 in e2e-models-e2e-worker-w0-ea5e33 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"simulator-w0-ea5e33","namespace":"e2e-models-e2e-worker-w0-ea5e33"},"spec":{"model":{"name":"e2e/simulator-w0-ea5e33","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-ea5e33","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/simulator-w0-ea5e33","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T21:17:12Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'simulator-w0-ea5e33', 'namespace': 'e2e-models-e2e-worker-w0-ea5e33', 'resourceVersion': '28372', 'uid': 'f4f0a941-6134-4b27-b5e7-fbb9080e60cf'}, 'spec': {'model': {'name': 'e2e/simulator-w0-ea5e33', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/simulator-w0-ea5e33', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-ea5e33/models/e2e/simulator-w0-ea5e33'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-ea5e33/models/e2e/simulator-w0-ea5e33'}, {'name': 'e2e/simulator-w0-ea5e33'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'simulator-w0-ea5e33-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-ea5e33'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'simulator-w0-ea5e33-kserve'}, 'service': {'kind': 'Service', 'name': 'simulator-w0-ea5e33-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_api_keys_context' for <Function test_create_api_key>> @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:253: in bootstrap_worker_tenant wait_for_llmisvc_backend_ready(model_ref, context.model_namespace, context.gateway_name) test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'simulator-w0-ea5e33' namespace = 'e2e-models-e2e-worker-w0-ea5e33' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/simulator-w0-ea5e33 in e2e-models-e2e-worker-w0-ea5e33 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"simulator-w0-ea5e33","namespace":"e2e-models-e2e-worker-w0-ea5e33"},"spec":{"model":{"name":"e2e/simulator-w0-ea5e33","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-ea5e33","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/simulator-w0-ea5e33","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T21:17:12Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'simulator-w0-ea5e33', 'namespace': 'e2e-models-e2e-worker-w0-ea5e33', 'resourceVersion': '28372', 'uid': 'f4f0a941-6134-4b27-b5e7-fbb9080e60cf'}, 'spec': {'model': {'name': 'e2e/simulator-w0-ea5e33', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/simulator-w0-ea5e33', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-ea5e33/models/e2e/simulator-w0-ea5e33'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-ea5e33/models/e2e/simulator-w0-ea5e33'}, {'name': 'e2e/simulator-w0-ea5e33'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'simulator-w0-ea5e33-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-ea5e33'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'simulator-w0-ea5e33-kserve'}, 'service': {'kind': 'Service', 'name': 'simulator-w0-ea5e33-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionErrorrequest = <SubRequest '_worker_api_keys_context' for <Function test_create_api_key>> @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:253: in bootstrap_worker_tenant wait_for_llmisvc_backend_ready(model_ref, context.model_namespace, context.gateway_name) test/e2e/tests/multitenancy_helpers.py:706: in wait_for_llmisvc_backend_ready llmisvc = wait_for_status_condition( test/e2e/tests/multitenancy_helpers.py:303: in wait_for_status_condition return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'llminferenceservice', name = 'simulator-w0-ea5e33' namespace = 'e2e-models-e2e-worker-w0-ea5e33' def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or '<cluster>'} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: llminferenceservice/simulator-w0-ea5e33 in e2e-models-e2e-worker-w0-ea5e33 did not satisfy condition. Last object: {'apiVersion': 'serving.kserve.io/v1alpha2', 'kind': 'LLMInferenceService', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"serving.kserve.io/v1alpha1","kind":"LLMInferenceService","metadata":{"annotations":{},"name":"simulator-w0-ea5e33","namespace":"e2e-models-e2e-worker-w0-ea5e33"},"spec":{"model":{"name":"e2e/simulator-w0-ea5e33","uri":"hf://placeholder/no-model"},"replicas":1,"router":{"gateway":{"refs":[{"name":"e2e-worker-w0-ea5e33","namespace":"openshift-ingress"}]},"route":{}},"storageInitializer":{"enabled":false},"template":{"containers":[{"args":["--port","8000","--model","e2e/simulator-w0-ea5e33","--mode","random","--no-mm-encoder-only","--ssl-certfile","/var/run/kserve/tls/tls.crt","--ssl-keyfile","/var/run/kserve/tls/tls.key"],"command":["/app/llm-d-inference-sim"],"image":"ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1","livenessProbe":{"httpGet":{"path":"/health","port":"https","scheme":"HTTPS"}},"name":"main","ports":[{"containerPort":8000,"name":"https","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"https","scheme":"HTTPS"}}}]}}}\n'}, 'creationTimestamp': '2026-09-11T21:17:12Z', 'finalizers': ['serving.kserve.io/llmisvc-finalizer'], 'generation': 1, 'name': 'simulator-w0-ea5e33', 'namespace': 'e2e-models-e2e-worker-w0-ea5e33', 'resourceVersion': '28372', 'uid': 'f4f0a941-6134-4b27-b5e7-fbb9080e60cf'}, 'spec': {'model': {'name': 'e2e/simulator-w0-ea5e33', 'uri': 'hf://placeholder/no-model'}, 'replicas': 1, 'router': {'gateway': {'refs': [{'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}]}, 'route': {}}, 'storageInitializer': {'enabled': False}, 'template': {'containers': [{'args': ['--port', '8000', '--model', 'e2e/simulator-w0-ea5e33', '--mode', 'random', '--no-mm-encoder-only', '--ssl-certfile', '/var/run/kserve/tls/tls.crt', '--ssl-keyfile', '/var/run/kserve/tls/tls.key'], 'command': ['/app/llm-d-inference-sim'], 'image': 'ghcr.io/llm-d/llm-d-inference-sim@sha256:c3ba435081a4d032676b218ea34eb3a1c54507da0fade2f6297f9c37894fe0d1', 'livenessProbe': {'httpGet': {'path': '/health', 'port': 'https', 'scheme': 'HTTPS'}}, 'name': 'main', 'ports': [{'containerPort': 8000, 'name': 'https', 'protocol': 'TCP'}], 'readinessProbe': {'httpGet': {'path': '/ready', 'port': 'https', 'scheme': 'HTTPS'}}, 'resources': {}}]}}, 'status': {'addresses': [{'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-ea5e33/models/e2e/simulator-w0-ea5e33'}], 'name': 'gateway-internal-model-routing', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/'}, {'models': [{'name': 'publishers/e2e-models-e2e-worker-w0-ea5e33/models/e2e/simulator-w0-ea5e33'}, {'name': 'e2e/simulator-w0-ea5e33'}], 'name': 'gateway-internal', 'origin': {'group': 'gateway.networking.k8s.io', 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33'}], 'annotations': {'serving.kserve.io/config-llm-decode-template': 'v3-5-0-ea-2-kserve-config-llm-decode-template', 'serving.kserve.io/config-llm-decode-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-decode-worker-data-parallel', 'serving.kserve.io/config-llm-prefill-template': 'v3-5-0-ea-2-kserve-config-llm-prefill-template', 'serving.kserve.io/config-llm-prefill-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-prefill-worker-data-parallel', 'serving.kserve.io/config-llm-router-route': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'serving.kserve.io/config-llm-scheduler': 'v3-5-0-ea-2-kserve-config-llm-scheduler', 'serving.kserve.io/config-llm-template': 'v3-5-0-ea-2-kserve-config-llm-template', 'serving.kserve.io/config-llm-worker-data-parallel': 'v3-5-0-ea-2-kserve-config-llm-worker-data-parallel'}, 'appliedConfigs': [{'name': 'v3-5-0-ea-2-kserve-config-llm-router-route', 'namespace': 'opendatahub', 'source': 'Preset'}, {'name': 'v3-5-0-ea-2-kserve-config-llm-template', 'namespace': 'opendatahub', 'source': 'Preset'}], 'conditions': [{'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-11T21:17:55Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}], 'observedGeneration': 1, 'router': {'gateways': [{'group': 'gateway.networking.k8s.io', 'httpRoutes': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute', 'name': 'simulator-w0-ea5e33-kserve-route', 'namespace': 'e2e-models-e2e-worker-w0-ea5e33'}], 'kind': 'Gateway', 'name': 'e2e-worker-w0-ea5e33', 'namespace': 'openshift-ingress'}]}, 'url': 'https://e2e-worker-w0-ea5e33-openshift-default.openshift-ingress.svc.cluster.local/e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33', 'workloads': {'primary': {'apiGroup': 'apps', 'kind': 'Deployment', 'name': 'simulator-w0-ea5e33-kserve'}, 'service': {'kind': 'Service', 'name': 'simulator-w0-ea5e33-kserve-workload-svc'}}}} test/e2e/tests/multitenancy_helpers.py:229: AssertionError/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:189: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deploy/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:355: Dormant-mode test mutates controller flags; set ENABLE_TENANT_DISCOVERY_DORMANT_E2E=true/workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:215: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B