/workspace/source/test/e2e/tests/test_namespace_scoping.py:214: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:247: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:287: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:324: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:382: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_namespace_scoping.py:456: test_namespace_scoping validates single-tenant dormant mode; skipped when ENABLE_TENANT_NAMESPACE_DISCOVERY=true/workspace/source/test/e2e/tests/test_tenant_discovery.py:79: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery.py:155: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_external_oidc.py:252: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:263: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:277: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:290: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:303: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:335: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:379: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:395: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:405: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:428: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:436: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:441: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:454: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:501: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:537: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:602: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:633: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:659: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:690: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:750: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:786: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:824: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:872: EXTERNAL_OIDC is not true/workspace/source/test/e2e/tests/test_external_oidc.py:961: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1013: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1049: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1103: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_external_oidc.py:1142: EXTERNAL_OIDC not enabled/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:189: OIDC_ISSUER_URL not set; per-tenant OIDC E2E requires external OIDC deploy/workspace/source/test/e2e/tests/test_tenant_namespace_discovery.py:355: Dormant-mode test mutates controller flags; set ENABLE_TENANT_DISCOVERY_DORMANT_E2E=true/workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:93: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:141: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_discovery_isolation.py:228: Skipping when Gateway uses ClusterIP + OpenShift Route (unsupported configuration). This mixes incompatible routing paradigms. Gateway has no external hostname in spec.listeners, so /v1/tenants returns an error. Supported configuration: LoadBalancer service with hostname in spec.listeners./workspace/source/test/e2e/tests/test_tenant_auth_isolation.py:215: Per-tenant OIDC tokens unavailable — set OIDC_TOKEN_URL (tenant-a) and OIDC_TOKEN_URL_TENANT_B (tenant-b), or OIDC_TOKEN_TENANT_A / OIDC_TOKEN_TENANT_B@pytest.fixture(scope="session") def worker_tenant_context(): """Create one isolated AITenant for tests explicitly marked worker_tenant.""" from worker_tenant_fixtures import ( bootstrap_worker_tenant, build_worker_tenant_case, teardown_worker_tenant, worker_tenant_enabled, ) if not worker_tenant_enabled(): pytest.skip("worker tenant isolation disabled by E2E_USE_WORKER_TENANT") case = build_worker_tenant_case(_xdist_worker_suffix()) try: > case = bootstrap_worker_tenant(case) test/e2e/tests/conftest.py:37: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/tests/worker_tenant_fixtures.py:287: in bootstrap_worker_tenant _wait_for_model_ready( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_ref = 'distinct-w4-1312ea', namespace = 'e2e-models-e2e-worker-w4-1312ea' timeout = 180 def _wait_for_model_ready(model_ref, namespace=MODEL_NAMESPACE, timeout=60): """Wait for MaaSModelRef to reach Ready phase. Args: model_ref: Name of the MaaSModelRef namespace: Namespace (default: MODEL_NAMESPACE) timeout: Maximum wait time in seconds (default: 60) Returns: The MaaSModelRef CR dict when Ready Raises: TimeoutError: If MaaSModelRef doesn't reach Ready within timeout """ deadline = time.time() + timeout log.info(f"Waiting for MaaSModelRef {namespace}/{model_ref} to reach phase 'Ready' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maasmodelref", model_ref, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") endpoint = status.get("endpoint") if phase == "Ready" and endpoint: log.info(f"MaaSModelRef {namespace}/{model_ref} is Ready with endpoint: {endpoint}") return cr log.debug(f"MaaSModelRef {namespace}/{model_ref}: phase={phase}, endpoint={endpoint or 'none'}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maasmodelref", model_ref, namespace) status = cr.get("status", {}) if cr else {} conditions = status.get("conditions", []) > raise TimeoutError( f"MaaSModelRef {namespace}/{model_ref} did not reach Ready within {timeout}s " f"(current: phase={status.get('phase')}, endpoint={status.get('endpoint')}, " f"conditions={[c.get('type') + '=' + str(c.get('status')) for c in conditions]})" ) E TimeoutError: MaaSModelRef e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea did not reach Ready within 180s (current: phase=Pending, endpoint=None, conditions=['GovernanceAttached=False', 'RuntimeReady=True', 'ModelIdentityUnique=True', 'Ready=False']) test/e2e/tests/test_helper.py:1499: TimeoutErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_models_endpoint.py:88: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1108: in execute raise exc.with_traceback(exc_tb) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:287: in bootstrap_worker_tenant _wait_for_model_ready( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_ref = 'distinct-w4-1312ea', namespace = 'e2e-models-e2e-worker-w4-1312ea' timeout = 180 def _wait_for_model_ready(model_ref, namespace=MODEL_NAMESPACE, timeout=60): """Wait for MaaSModelRef to reach Ready phase. Args: model_ref: Name of the MaaSModelRef namespace: Namespace (default: MODEL_NAMESPACE) timeout: Maximum wait time in seconds (default: 60) Returns: The MaaSModelRef CR dict when Ready Raises: TimeoutError: If MaaSModelRef doesn't reach Ready within timeout """ deadline = time.time() + timeout log.info(f"Waiting for MaaSModelRef {namespace}/{model_ref} to reach phase 'Ready' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maasmodelref", model_ref, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") endpoint = status.get("endpoint") if phase == "Ready" and endpoint: log.info(f"MaaSModelRef {namespace}/{model_ref} is Ready with endpoint: {endpoint}") return cr log.debug(f"MaaSModelRef {namespace}/{model_ref}: phase={phase}, endpoint={endpoint or 'none'}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maasmodelref", model_ref, namespace) status = cr.get("status", {}) if cr else {} conditions = status.get("conditions", []) > raise TimeoutError( f"MaaSModelRef {namespace}/{model_ref} did not reach Ready within {timeout}s " f"(current: phase={status.get('phase')}, endpoint={status.get('endpoint')}, " f"conditions={[c.get('type') + '=' + str(c.get('status')) for c in conditions]})" ) E TimeoutError: MaaSModelRef e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea did not reach Ready within 180s (current: phase=Pending, endpoint=None, conditions=['GovernanceAttached=False', 'RuntimeReady=True', 'ModelIdentityUnique=True', 'Ready=False']) test/e2e/tests/test_helper.py:1499: TimeoutErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_models_endpoint.py:88: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1108: in execute raise exc.with_traceback(exc_tb) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:287: in bootstrap_worker_tenant _wait_for_model_ready( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_ref = 'distinct-w4-1312ea', namespace = 'e2e-models-e2e-worker-w4-1312ea' timeout = 180 def _wait_for_model_ready(model_ref, namespace=MODEL_NAMESPACE, timeout=60): """Wait for MaaSModelRef to reach Ready phase. Args: model_ref: Name of the MaaSModelRef namespace: Namespace (default: MODEL_NAMESPACE) timeout: Maximum wait time in seconds (default: 60) Returns: The MaaSModelRef CR dict when Ready Raises: TimeoutError: If MaaSModelRef doesn't reach Ready within timeout """ deadline = time.time() + timeout log.info(f"Waiting for MaaSModelRef {namespace}/{model_ref} to reach phase 'Ready' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maasmodelref", model_ref, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") endpoint = status.get("endpoint") if phase == "Ready" and endpoint: log.info(f"MaaSModelRef {namespace}/{model_ref} is Ready with endpoint: {endpoint}") return cr log.debug(f"MaaSModelRef {namespace}/{model_ref}: phase={phase}, endpoint={endpoint or 'none'}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maasmodelref", model_ref, namespace) status = cr.get("status", {}) if cr else {} conditions = status.get("conditions", []) > raise TimeoutError( f"MaaSModelRef {namespace}/{model_ref} did not reach Ready within {timeout}s " f"(current: phase={status.get('phase')}, endpoint={status.get('endpoint')}, " f"conditions={[c.get('type') + '=' + str(c.get('status')) for c in conditions]})" ) E TimeoutError: MaaSModelRef e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea did not reach Ready within 180s (current: phase=Pending, endpoint=None, conditions=['GovernanceAttached=False', 'RuntimeReady=True', 'ModelIdentityUnique=True', 'Ready=False']) test/e2e/tests/test_helper.py:1499: TimeoutErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_models_endpoint.py:88: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1108: in execute raise exc.with_traceback(exc_tb) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:287: in bootstrap_worker_tenant _wait_for_model_ready( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_ref = 'distinct-w4-1312ea', namespace = 'e2e-models-e2e-worker-w4-1312ea' timeout = 180 def _wait_for_model_ready(model_ref, namespace=MODEL_NAMESPACE, timeout=60): """Wait for MaaSModelRef to reach Ready phase. Args: model_ref: Name of the MaaSModelRef namespace: Namespace (default: MODEL_NAMESPACE) timeout: Maximum wait time in seconds (default: 60) Returns: The MaaSModelRef CR dict when Ready Raises: TimeoutError: If MaaSModelRef doesn't reach Ready within timeout """ deadline = time.time() + timeout log.info(f"Waiting for MaaSModelRef {namespace}/{model_ref} to reach phase 'Ready' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maasmodelref", model_ref, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") endpoint = status.get("endpoint") if phase == "Ready" and endpoint: log.info(f"MaaSModelRef {namespace}/{model_ref} is Ready with endpoint: {endpoint}") return cr log.debug(f"MaaSModelRef {namespace}/{model_ref}: phase={phase}, endpoint={endpoint or 'none'}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maasmodelref", model_ref, namespace) status = cr.get("status", {}) if cr else {} conditions = status.get("conditions", []) > raise TimeoutError( f"MaaSModelRef {namespace}/{model_ref} did not reach Ready within {timeout}s " f"(current: phase={status.get('phase')}, endpoint={status.get('endpoint')}, " f"conditions={[c.get('type') + '=' + str(c.get('status')) for c in conditions]})" ) E TimeoutError: MaaSModelRef e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea did not reach Ready within 180s (current: phase=Pending, endpoint=None, conditions=['GovernanceAttached=False', 'RuntimeReady=True', 'ModelIdentityUnique=True', 'Ready=False']) test/e2e/tests/test_helper.py:1499: TimeoutErrorrequest = <SubRequest '_worker_models_context' for <Function test_explicit_subscription_header>> @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_models_endpoint.py:88: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1108: in execute raise exc.with_traceback(exc_tb) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:287: in bootstrap_worker_tenant _wait_for_model_ready( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_ref = 'distinct-w4-1312ea', namespace = 'e2e-models-e2e-worker-w4-1312ea' timeout = 180 def _wait_for_model_ready(model_ref, namespace=MODEL_NAMESPACE, timeout=60): """Wait for MaaSModelRef to reach Ready phase. Args: model_ref: Name of the MaaSModelRef namespace: Namespace (default: MODEL_NAMESPACE) timeout: Maximum wait time in seconds (default: 60) Returns: The MaaSModelRef CR dict when Ready Raises: TimeoutError: If MaaSModelRef doesn't reach Ready within timeout """ deadline = time.time() + timeout log.info(f"Waiting for MaaSModelRef {namespace}/{model_ref} to reach phase 'Ready' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maasmodelref", model_ref, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") endpoint = status.get("endpoint") if phase == "Ready" and endpoint: log.info(f"MaaSModelRef {namespace}/{model_ref} is Ready with endpoint: {endpoint}") return cr log.debug(f"MaaSModelRef {namespace}/{model_ref}: phase={phase}, endpoint={endpoint or 'none'}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maasmodelref", model_ref, namespace) status = cr.get("status", {}) if cr else {} conditions = status.get("conditions", []) > raise TimeoutError( f"MaaSModelRef {namespace}/{model_ref} did not reach Ready within {timeout}s " f"(current: phase={status.get('phase')}, endpoint={status.get('endpoint')}, " f"conditions={[c.get('type') + '=' + str(c.get('status')) for c in conditions]})" ) E TimeoutError: MaaSModelRef e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea did not reach Ready within 180s (current: phase=Pending, endpoint=None, conditions=['GovernanceAttached=False', 'RuntimeReady=True', 'ModelIdentityUnique=True', 'Ready=False']) test/e2e/tests/test_helper.py:1499: TimeoutErrorrequest = <SubRequest '_worker_api_keys_context' for <Function test_create_api_key>> @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:287: in bootstrap_worker_tenant _wait_for_model_ready( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_ref = 'distinct-w0-30919a', namespace = 'e2e-models-e2e-worker-w0-30919a' timeout = 180 def _wait_for_model_ready(model_ref, namespace=MODEL_NAMESPACE, timeout=60): """Wait for MaaSModelRef to reach Ready phase. Args: model_ref: Name of the MaaSModelRef namespace: Namespace (default: MODEL_NAMESPACE) timeout: Maximum wait time in seconds (default: 60) Returns: The MaaSModelRef CR dict when Ready Raises: TimeoutError: If MaaSModelRef doesn't reach Ready within timeout """ deadline = time.time() + timeout log.info(f"Waiting for MaaSModelRef {namespace}/{model_ref} to reach phase 'Ready' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maasmodelref", model_ref, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") endpoint = status.get("endpoint") if phase == "Ready" and endpoint: log.info(f"MaaSModelRef {namespace}/{model_ref} is Ready with endpoint: {endpoint}") return cr log.debug(f"MaaSModelRef {namespace}/{model_ref}: phase={phase}, endpoint={endpoint or 'none'}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maasmodelref", model_ref, namespace) status = cr.get("status", {}) if cr else {} conditions = status.get("conditions", []) > raise TimeoutError( f"MaaSModelRef {namespace}/{model_ref} did not reach Ready within {timeout}s " f"(current: phase={status.get('phase')}, endpoint={status.get('endpoint')}, " f"conditions={[c.get('type') + '=' + str(c.get('status')) for c in conditions]})" ) E TimeoutError: MaaSModelRef e2e-models-e2e-worker-w0-30919a/distinct-w0-30919a did not reach Ready within 180s (current: phase=Pending, endpoint=None, conditions=['GovernanceAttached=False', 'RuntimeReady=True', 'ModelIdentityUnique=True', 'Ready=False']) test/e2e/tests/test_helper.py:1499: TimeoutErrorrequest = <SubRequest '_worker_api_keys_context' for <Function test_create_api_key>> @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:287: in bootstrap_worker_tenant _wait_for_model_ready( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_ref = 'distinct-w0-30919a', namespace = 'e2e-models-e2e-worker-w0-30919a' timeout = 180 def _wait_for_model_ready(model_ref, namespace=MODEL_NAMESPACE, timeout=60): """Wait for MaaSModelRef to reach Ready phase. Args: model_ref: Name of the MaaSModelRef namespace: Namespace (default: MODEL_NAMESPACE) timeout: Maximum wait time in seconds (default: 60) Returns: The MaaSModelRef CR dict when Ready Raises: TimeoutError: If MaaSModelRef doesn't reach Ready within timeout """ deadline = time.time() + timeout log.info(f"Waiting for MaaSModelRef {namespace}/{model_ref} to reach phase 'Ready' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maasmodelref", model_ref, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") endpoint = status.get("endpoint") if phase == "Ready" and endpoint: log.info(f"MaaSModelRef {namespace}/{model_ref} is Ready with endpoint: {endpoint}") return cr log.debug(f"MaaSModelRef {namespace}/{model_ref}: phase={phase}, endpoint={endpoint or 'none'}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maasmodelref", model_ref, namespace) status = cr.get("status", {}) if cr else {} conditions = status.get("conditions", []) > raise TimeoutError( f"MaaSModelRef {namespace}/{model_ref} did not reach Ready within {timeout}s " f"(current: phase={status.get('phase')}, endpoint={status.get('endpoint')}, " f"conditions={[c.get('type') + '=' + str(c.get('status')) for c in conditions]})" ) E TimeoutError: MaaSModelRef e2e-models-e2e-worker-w0-30919a/distinct-w0-30919a did not reach Ready within 180s (current: phase=Pending, endpoint=None, conditions=['GovernanceAttached=False', 'RuntimeReady=True', 'ModelIdentityUnique=True', 'Ready=False']) test/e2e/tests/test_helper.py:1499: TimeoutErrorrequest = <SubRequest '_worker_api_keys_context' for <Function test_create_api_key>> @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:287: in bootstrap_worker_tenant _wait_for_model_ready( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_ref = 'distinct-w0-30919a', namespace = 'e2e-models-e2e-worker-w0-30919a' timeout = 180 def _wait_for_model_ready(model_ref, namespace=MODEL_NAMESPACE, timeout=60): """Wait for MaaSModelRef to reach Ready phase. Args: model_ref: Name of the MaaSModelRef namespace: Namespace (default: MODEL_NAMESPACE) timeout: Maximum wait time in seconds (default: 60) Returns: The MaaSModelRef CR dict when Ready Raises: TimeoutError: If MaaSModelRef doesn't reach Ready within timeout """ deadline = time.time() + timeout log.info(f"Waiting for MaaSModelRef {namespace}/{model_ref} to reach phase 'Ready' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maasmodelref", model_ref, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") endpoint = status.get("endpoint") if phase == "Ready" and endpoint: log.info(f"MaaSModelRef {namespace}/{model_ref} is Ready with endpoint: {endpoint}") return cr log.debug(f"MaaSModelRef {namespace}/{model_ref}: phase={phase}, endpoint={endpoint or 'none'}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maasmodelref", model_ref, namespace) status = cr.get("status", {}) if cr else {} conditions = status.get("conditions", []) > raise TimeoutError( f"MaaSModelRef {namespace}/{model_ref} did not reach Ready within {timeout}s " f"(current: phase={status.get('phase')}, endpoint={status.get('endpoint')}, " f"conditions={[c.get('type') + '=' + str(c.get('status')) for c in conditions]})" ) E TimeoutError: MaaSModelRef e2e-models-e2e-worker-w0-30919a/distinct-w0-30919a did not reach Ready within 180s (current: phase=Pending, endpoint=None, conditions=['GovernanceAttached=False', 'RuntimeReady=True', 'ModelIdentityUnique=True', 'Ready=False']) test/e2e/tests/test_helper.py:1499: TimeoutErrorrequest = <SubRequest '_worker_api_keys_context' for <Function test_create_api_key>> @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:287: in bootstrap_worker_tenant _wait_for_model_ready( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_ref = 'distinct-w0-30919a', namespace = 'e2e-models-e2e-worker-w0-30919a' timeout = 180 def _wait_for_model_ready(model_ref, namespace=MODEL_NAMESPACE, timeout=60): """Wait for MaaSModelRef to reach Ready phase. Args: model_ref: Name of the MaaSModelRef namespace: Namespace (default: MODEL_NAMESPACE) timeout: Maximum wait time in seconds (default: 60) Returns: The MaaSModelRef CR dict when Ready Raises: TimeoutError: If MaaSModelRef doesn't reach Ready within timeout """ deadline = time.time() + timeout log.info(f"Waiting for MaaSModelRef {namespace}/{model_ref} to reach phase 'Ready' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maasmodelref", model_ref, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") endpoint = status.get("endpoint") if phase == "Ready" and endpoint: log.info(f"MaaSModelRef {namespace}/{model_ref} is Ready with endpoint: {endpoint}") return cr log.debug(f"MaaSModelRef {namespace}/{model_ref}: phase={phase}, endpoint={endpoint or 'none'}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maasmodelref", model_ref, namespace) status = cr.get("status", {}) if cr else {} conditions = status.get("conditions", []) > raise TimeoutError( f"MaaSModelRef {namespace}/{model_ref} did not reach Ready within {timeout}s " f"(current: phase={status.get('phase')}, endpoint={status.get('endpoint')}, " f"conditions={[c.get('type') + '=' + str(c.get('status')) for c in conditions]})" ) E TimeoutError: MaaSModelRef e2e-models-e2e-worker-w0-30919a/distinct-w0-30919a did not reach Ready within 180s (current: phase=Pending, endpoint=None, conditions=['GovernanceAttached=False', 'RuntimeReady=True', 'ModelIdentityUnique=True', 'Ready=False']) test/e2e/tests/test_helper.py:1499: TimeoutErrorrequest = <SubRequest '_worker_api_keys_context' for <Function test_create_api_key>> @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/e2e/tests/worker_tenant_fixtures.py:287: in bootstrap_worker_tenant _wait_for_model_ready( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_ref = 'distinct-w0-30919a', namespace = 'e2e-models-e2e-worker-w0-30919a' timeout = 180 def _wait_for_model_ready(model_ref, namespace=MODEL_NAMESPACE, timeout=60): """Wait for MaaSModelRef to reach Ready phase. Args: model_ref: Name of the MaaSModelRef namespace: Namespace (default: MODEL_NAMESPACE) timeout: Maximum wait time in seconds (default: 60) Returns: The MaaSModelRef CR dict when Ready Raises: TimeoutError: If MaaSModelRef doesn't reach Ready within timeout """ deadline = time.time() + timeout log.info(f"Waiting for MaaSModelRef {namespace}/{model_ref} to reach phase 'Ready' (timeout: {timeout}s)...") while time.time() < deadline: cr = _get_cr("maasmodelref", model_ref, namespace) if cr: status = cr.get("status", {}) phase = status.get("phase") endpoint = status.get("endpoint") if phase == "Ready" and endpoint: log.info(f"MaaSModelRef {namespace}/{model_ref} is Ready with endpoint: {endpoint}") return cr log.debug(f"MaaSModelRef {namespace}/{model_ref}: phase={phase}, endpoint={endpoint or 'none'}") time.sleep(2) # Timeout - return current state for debugging cr = _get_cr("maasmodelref", model_ref, namespace) status = cr.get("status", {}) if cr else {} conditions = status.get("conditions", []) > raise TimeoutError( f"MaaSModelRef {namespace}/{model_ref} did not reach Ready within {timeout}s " f"(current: phase={status.get('phase')}, endpoint={status.get('endpoint')}, " f"conditions={[c.get('type') + '=' + str(c.get('status')) for c in conditions]})" ) E TimeoutError: MaaSModelRef e2e-models-e2e-worker-w0-30919a/distinct-w0-30919a did not reach Ready within 180s (current: phase=Pending, endpoint=None, conditions=['GovernanceAttached=False', 'RuntimeReady=True', 'ModelIdentityUnique=True', 'Ready=False']) test/e2e/tests/test_helper.py:1499: TimeoutError