--- pytest attempt 2026-09-15T16:26:47.587311+00:00 (/.local/bin/uv --directory /home/odh/opendatahub-tests...) --- 2026-09-15T16:26:50.023186Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-09-15T16:26:50.023437Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.rh-nightly-pm.nuth.p3.openshiftapps.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.rh-nightly-pm.nuth.p3.openshiftapps.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.rh-nightly-pm.nuth.p3.openshiftapps.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.rh-nightly-pm.nuth.p3.openshiftapps.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.rh-nightly-pm.nuth.p3.openshiftapps.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-09-15T16:26:50.158280Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.rh-nightly-pm.nuth.p3.openshiftapps.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.rh-nightly-pm.nuth.p3.openshiftapps.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.rh-nightly-pm.nuth.p3.openshiftapps.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.rh-nightly-pm.nuth.p3.openshiftapps.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-09-15T16:26:50.226019Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.rh-nightly-pm.nuth.p3.openshiftapps.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( ============================= test session starts ============================== platform linux -- Python 3.14.7, pytest-9.1.1, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python3 cachedir: .pytest_cache metadata: {'Python': '3.14.7', 'Platform': 'Linux-5.14.0-570.132.1.el9_6.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.1.1', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.14.2', 'langsmith': '0.10.10', 'asyncio': '1.4.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.5.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.5.3'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.14.2, langsmith-0.10.10, asyncio-1.4.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.5.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.5.3 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... collected 304 items / 298 deselected / 6 selected tests/ai_safety/evalhub/single_tenancy/test_evalhub_single_tenancy_k8s.py::TestEvalHubSingleTenancyOperatorResources::test_deployment_created_and_ready[model_namespace0]  --------------------------------- test_deployment_created_and_ready[model_namespace0] --------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-09-15T16:26:51.009867Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-09-15T16:26:51.012081Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-09-15T16:26:51.012274Z [info ] Executing session fixture: admin_client [conftest] 2026-09-15T16:26:51.012487Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-09-15T16:26:51.013803Z [info ] Executing session fixture: dsc_resource [conftest] 2026-09-15T16:26:51.013950Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-09-15T16:26:51.079659Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-09-15T16:26:51.079877Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-09-15T16:26:51.080942Z [info ] Executing session fixture: os_path_environment [conftest] 2026-09-15T16:26:51.081112Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-09-15T16:26:51.081271Z [info ] Executing session fixture: bin_directory [conftest] 2026-09-15T16:26:51.082673Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-09-15T16:26:51.081765Z [warning ] Using previously installed: /workspace/tests-shared/tests-payload/.tools/bin/oc [tests.conftest] name=tests.conftest 2026-09-15T16:26:51.083016Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-09-15T16:26:51.081905Z [info ] OC binary path: /workspace/tests-shared/tests-payload/.tools/bin/oc [tests.conftest] name=tests.conftest 2026-09-15T16:26:51.081951Z [info ] Adding /tmp/pytest-4nLKaBKNGsv93qJvKBHxAe/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-09-15T16:26:51.083490Z [info ] Executing session fixture: nodes [conftest] 2026-09-15T16:26:52.089858Z [info ] Executing session fixture: dsci_resource [conftest] 2026-09-15T16:26:52.090051Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-09-15T16:26:52.105572Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-09-15T16:26:52.105824Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-09-15T16:26:52.105965Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-09-15T16:26:52.104876Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-09-15T16:26:52.106192Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-09-15T16:26:52.135386Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-09-15T16:26:52.164300Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsci_status_ready Kwargs: {'dsci_resource': }) [timeout_sampler] 2026-09-15T16:26:52.163765Z [info ] Wait for DSCI default-dsci to be in Ready status. [utilities.infra] name=utilities.infra 2026-09-15T16:26:52.164689Z [info ] Get DSCInitialization default-dsci status [ocp_resources DSCInitialization] 2026-09-15T16:26:52.178893Z [success ] Function: utilities.infra.wait_for_dsci_status_ready succeeded after 8.559226989746094e-05 [0:00:00.000086]. [timeout_sampler] 2026-09-15T16:26:52.179400Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': }) [timeout_sampler] 2026-09-15T16:26:52.178367Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-09-15T16:26:52.179977Z [info ] Get DataScienceCluster default-dsc status [ocp_resources DataScienceCluster] 2026-09-15T16:26:52.199632Z [success ] Function: utilities.infra.wait_for_dsc_status_ready succeeded after 5.602836608886719e-05 [0:00:00.000056]. [timeout_sampler] 2026-09-15T16:26:52.199821Z [info ] Executing session fixture: autouse_fixtures [conftest] 2026-09-15T16:26:52.199987Z [info ] Executing session fixture: pytestconfig [conftest] 2026-09-15T16:26:52.200140Z [info ] Executing session fixture: teardown_resources [conftest] 2026-09-15T16:26:52.200267Z [info ] Executing class fixture: model_namespace [conftest] 2026-09-15T16:26:52.200390Z [info ] Create Namespace test-evalhub-st-resources [ocp_resources Namespace] 2026-09-15T16:26:52.200544Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'test-evalhub-st-resources'}, 'spec': {}} [ocp_resources Namespace] 2026-09-15T16:26:52.248106Z [info ] Wait for Namespace test-evalhub-st-resources status to be Active [ocp_resources Namespace] 2026-09-15T16:26:52.248258Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-09-15T16:26:52.274954Z [info ] Status of Namespace test-evalhub-st-resources is Active [ocp_resources Namespace] 2026-09-15T16:26:52.275166Z [success ] Function: lambda: self.exists succeeded after 5.0067901611328125e-05 [0:00:00.000050]. [timeout_sampler] 2026-09-15T16:26:52.275304Z [info ] Executing class fixture: evalhub_st_cr [conftest] 2026-09-15T16:26:52.275435Z [info ] kind: CustomResourceDefinition api version: apiextensions.k8s.io/v1 [ocp_resources.resource] 2026-09-15T16:26:52.338924Z [info ] kind: EvalHub api version: trustyai.opendatahub.io/v1 [ocp_resources.resource] 2026-09-15T16:26:52.339117Z [info ] Create EvalHub evalhub-st  [ocp_resources EvalHub] 2026-09-15T16:26:52.339258Z [info ] Posting {'apiVersion': 'trustyai.opendatahub.io/v1', 'kind': 'EvalHub', 'metadata': {'name': 'evalhub-st', 'namespace': 'test-evalhub-st-resources'}, 'spec': {'database': {'type': 'sqlite'}, 'tenancy': 'single'}} [ocp_resources EvalHub] 2026-09-15T16:26:52.377906Z [info ] Wait until EvalHub evalhub-st is created [ocp_resources EvalHub] 2026-09-15T16:26:52.378084Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-09-15T16:26:52.394904Z [success ] Function: lambda: self.exists succeeded after 5.936622619628906e-05 [0:00:00.000059]. [timeout_sampler] 2026-09-15T16:26:52.395095Z [info ] Executing class fixture: evalhub_st_deployment [conftest] 2026-09-15T16:26:52.395227Z [info ] kind: Deployment api version: apps/v1 [ocp_resources.resource] 2026-09-15T16:26:52.395356Z [info ] Wait for Deployment evalhub-st to be deployed: True [ocp_resources Deployment] 2026-09-15T16:26:52.395506Z [info ] Waiting for 300 seconds [0:05:00], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-09-15T16:26:55.459595Z [success ] Function: lambda: self.exists succeeded after 3.049125909805298 [0:00:03.049126]. [timeout_sampler] 2026-09-15T16:26:55.459799Z [info ] Waiting for 296.93537950515747 seconds [0:04:56.935380], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-09-15T16:31:52.964990Z [error ] Function: lambda: self.instance timed out after 296.93537950515747 [0:04:56.935380]. [timeout_sampler] ERROR _ ERROR at setup of TestEvalHubSingleTenancyOperatorResources.test_deployment_created_and_ready[model_namespace0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 170, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 536, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1710, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 806, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 630, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1232, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 924, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1328, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1005, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/single_tenancy/conftest.py", line 70, in evalhub_st_deployment deployment.wait_for_replicas(timeout=Timeout.TIMEOUT_5MIN) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/deployment.py", line 371, in wait_for_replicas for sample in samples: ^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 346, in __iter__ raise TimeoutExpiredError( ...<3 lines>... ) timeout_sampler.TimeoutExpiredError: Timed Out: 296.93537950515747 Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance Last exception: N/A. Elapsed time: 296.93537950515747 [0:04:56.935380] _ 1 of 6 completed, 0 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 1 Error, 0 ReRun _ tests/ai_safety/evalhub/single_tenancy/test_evalhub_single_tenancy_k8s.py::TestEvalHubSingleTenancyOperatorResources::test_service_created_with_correct_port[model_namespace0]  TEST: TestEvalHubSingleTenancyOperatorResources.test_deployment_created_and_ready[model_namespace0] [setup] STATUS: ERROR 2026-09-15T16:31:52.977962Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 170, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 536, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1710, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 806, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 630, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1232, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 924, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1328, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1005, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/single_tenancy/conftest.py", line 70, in evalhub_st_deployment deployment.wait_for_replicas(timeout=Timeout.TIMEOUT_5MIN) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/deployment.py", line 371, in wait_for_replicas for sample in samples: ^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 346, in __iter__ raise TimeoutExpiredError( ...<3 lines>... ) timeout_sampler.TimeoutExpiredError: Timed Out: 296.93537950515747 Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance Last exception: N/A. Elapsed time: 296.93537950515747 [0:04:56.935380] [conftest] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ------------------------------- test_service_created_with_correct_port[model_namespace0] ------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- PASSED TEST: TestEvalHubSingleTenancyOperatorResources.test_service_created_with_correct_port[model_namespace0] STATUS: PASSED _ 2 of 6 completed, 1 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 1 Error, 0 ReRun _ tests/ai_safety/evalhub/single_tenancy/test_evalhub_single_tenancy_k8s.py::TestEvalHubSingleTenancyOperatorResources::test_route_created[model_namespace0] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ----------------------------------------- test_route_created[model_namespace0] ----------------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- ERROR _ ERROR at setup of TestEvalHubSingleTenancyOperatorResources.test_route_created[model_namespace0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 170, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 536, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1710, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 806, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 630, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1181, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1204, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1328, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1005, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/single_tenancy/conftest.py", line 70, in evalhub_st_deployment deployment.wait_for_replicas(timeout=Timeout.TIMEOUT_5MIN) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/deployment.py", line 371, in wait_for_replicas for sample in samples: ^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 346, in __iter__ raise TimeoutExpiredError( ...<3 lines>... ) timeout_sampler.TimeoutExpiredError: Timed Out: 296.93537950515747 Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance Last exception: N/A. Elapsed time: 296.93537950515747 [0:04:56.935380] _ 3 of 6 completed, 1 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 2 Error, 0 ReRun _ tests/ai_safety/evalhub/single_tenancy/test_evalhub_single_tenancy_k8s.py::TestEvalHubSingleTenancyOperatorResources::test_metrics_service_created[model_namespace0]  TEST: TestEvalHubSingleTenancyOperatorResources.test_route_created[model_namespace0] [setup] STATUS: ERROR 2026-09-15T16:31:53.027628Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 170, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 536, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1710, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 806, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 630, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1181, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1204, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1328, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1005, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/single_tenancy/conftest.py", line 70, in evalhub_st_deployment deployment.wait_for_replicas(timeout=Timeout.TIMEOUT_5MIN) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/deployment.py", line 371, in wait_for_replicas for sample in samples: ^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 346, in __iter__ raise TimeoutExpiredError( ...<3 lines>... ) timeout_sampler.TimeoutExpiredError: Timed Out: 296.93537950515747 Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance Last exception: N/A. Elapsed time: 296.93537950515747 [0:04:56.935380] [conftest] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ------------------------------------ test_metrics_service_created[model_namespace0] ------------------------------------ -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- PASSED _ 4 of 6 completed, 2 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 2 Error, 0 ReRun _ tests/ai_safety/evalhub/single_tenancy/test_evalhub_single_tenancy_k8s.py::TestEvalHubSingleTenancyOperatorResources::test_service_monitor_created[model_namespace0]  TEST: TestEvalHubSingleTenancyOperatorResources.test_metrics_service_created[model_namespace0] STATUS: PASSED ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ------------------------------------ test_service_monitor_created[model_namespace0] ------------------------------------ -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-15T16:31:53.100840Z [info ] kind: ServiceMonitor api version: monitoring.coreos.com/v1 [ocp_resources.resource] PASSED TEST: TestEvalHubSingleTenancyOperatorResources.test_service_monitor_created[model_namespace0] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-09-15T16:31:53.115955Z [info ] Delete EvalHub evalhub-st  [ocp_resources EvalHub] 2026-09-15T16:31:53.148717Z [info ] Deleting {'apiVersion': 'trustyai.opendatahub.io/v1', 'kind': 'EvalHub', 'metadata': {'creationTimestamp': '2026-09-15T16:26:52Z', 'finalizers': ['trustyai.opendatahub.io/evalhub-finalizer'], 'generation': 1, 'managedFields': [{'apiVersion': 'trustyai.opendatahub.io/v1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'.': {}, 'f:collections': {}, 'f:database': {'.': {}, 'f:maxIdleConns': {}, 'f:maxOpenConns': {}, 'f:type': {}}, 'f:providers': {}, 'f:replicas': {}, 'f:tenancy': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-09-15T16:26:52Z'}, {'apiVersion': 'trustyai.opendatahub.io/v1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"trustyai.opendatahub.io/evalhub-finalizer"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-09-15T16:26:52Z'}, {'apiVersion': 'trustyai.opendatahub.io/v1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:activeCollections': {}, 'f:activeProviders': {}, 'f:conditions': {}, 'f:lastUpdateTime': {}, 'f:mcp': {'.': {}, 'f:phase': {}, 'f:ready': {}}, 'f:phase': {}, 'f:ready': {}, 'f:replicas': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-09-15T16:31:30Z'}], 'name': 'evalhub-st', 'namespace': 'test-evalhub-st-resources', 'resourceVersion': '309215676', 'uid': '76e474c8-16ad-40ac-a2be-92818c9dcec7'}, 'spec': {'collections': ['leaderboard-v2', 'safety-and-fairness-v1', 'toxicity-and-ethical-principles'], 'database': {'maxIdleConns': 5, 'maxOpenConns': 25, 'type': 'sqlite'}, 'providers': ['garak', 'garak-kfp', 'lm-evaluation-harness'], 'replicas': 1, 'tenancy': 'single'}, 'status': {'activeCollections': ['leaderboard-v2', 'safety-and-fairness-v1', 'toxicity-and-ethical-principles'], 'activeProviders': ['garak', 'garak-kfp', 'lm-evaluation-harness'], 'conditions': [{'lastTransitionTime': '2026-09-15T16:31:30Z', 'message': 'Waiting for deployment to be ready (0/1 replicas ready)', 'reason': 'DeploymentNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-15T16:31:30Z', 'reason': 'MonitoringReady', 'status': 'False', 'type': 'MonitoringDegraded'}], 'lastUpdateTime': '2026-09-15T16:31:30Z', 'mcp': {'phase': 'Disabled', 'ready': False}, 'phase': 'Pending', 'ready': 'False', 'replicas': 1}} [ocp_resources EvalHub] 2026-09-15T16:31:53.169621Z [info ] Wait until EvalHub evalhub-st is deleted [ocp_resources EvalHub] 2026-09-15T16:31:53.169774Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-09-15T16:31:54.200807Z [success ] Function: lambda: self.exists succeeded after 1.0151386260986328 [0:00:01.015139]. [timeout_sampler] 2026-09-15T16:31:54.215543Z [info ] Delete Namespace test-evalhub-st-resources [ocp_resources Namespace] 2026-09-15T16:31:54.246875Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'test-evalhub-st-resources', 'uid': '86a85977-b5b3-4684-acdd-08b67678ccd4', 'resourceVersion': '309209314', 'creationTimestamp': '2026-09-15T16:26:52Z', 'labels': {'kubernetes.io/metadata.name': 'test-evalhub-st-resources', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c40,c0', 'openshift.io/sa.scc.supplemental-groups': '1001560000/10000', 'openshift.io/sa.scc.uid-range': '1001560000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-09-15T16:26:52Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-09-15T16:26:52Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-09-15T16:26:52Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-09-15T16:31:54.272930Z [info ] Wait until Namespace test-evalhub-st-resources is deleted [ocp_resources Namespace] 2026-09-15T16:31:54.273115Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] _ 5 of 6 completed, 3 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 2 Error, 0 ReRun _ tests/ai_safety/evalhub/test_evalhub_deployment.py::test_evalhub_crd_exists 2026-09-15T16:32:02.415535Z [success ] Function: lambda: self.exists succeeded after 8.123589754104614 [0:00:08.123590]. [timeout_sampler]  ----------------------------------------------- test_evalhub_crd_exists ----------------------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-15T16:32:02.416394Z [info ] kind: CustomResourceDefinition api version: apiextensions.k8s.io/v1 [ocp_resources.resource] PASSED TEST: test_evalhub_crd_exists STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- _ 6 of 6 completed, 4 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 2 Error, 0 ReRun _ ==== 4 passed, 298 deselected, 417 warnings, 2 errors in 312.22s (0:05:12) ===== ==================================== ERRORS ==================================== _ ERROR at setup of TestEvalHubSingleTenancyOperatorResources.test_deployment_created_and_ready[model_namespace0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 170, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 536, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1710, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 806, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 630, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1232, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 924, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1328, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1005, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/single_tenancy/conftest.py", line 70, in evalhub_st_deployment deployment.wait_for_replicas(timeout=Timeout.TIMEOUT_5MIN) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/deployment.py", line 371, in wait_for_replicas for sample in samples: ^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 346, in __iter__ raise TimeoutExpiredError( ...<3 lines>... ) timeout_sampler.TimeoutExpiredError: Timed Out: 296.93537950515747 Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance Last exception: N/A. Elapsed time: 296.93537950515747 [0:04:56.935380] _ ERROR at setup of TestEvalHubSingleTenancyOperatorResources.test_route_created[model_namespace0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 170, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 536, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1710, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 806, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 630, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1181, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1204, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1328, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1005, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/single_tenancy/conftest.py", line 70, in evalhub_st_deployment deployment.wait_for_replicas(timeout=Timeout.TIMEOUT_5MIN) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/deployment.py", line 371, in wait_for_replicas for sample in samples: ^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 346, in __iter__ raise TimeoutExpiredError( ...<3 lines>... ) timeout_sampler.TimeoutExpiredError: Timed Out: 296.93537950515747 Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance Last exception: N/A. Elapsed time: 296.93537950515747 [0:04:56.935380] =============================== warnings summary =============================== tests/ai_safety/evalhub/single_tenancy/test_evalhub_single_tenancy_k8s.py: 415 warnings tests/ai_safety/evalhub/test_evalhub_deployment.py: 2 warnings /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.rh-nightly-pm.nuth.p3.openshiftapps.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: /workspace/tests-shared/tests-payload/results/ai-safety-evalhub-smoke.xml - =========================== short test summary info ============================ ERROR tests/ai_safety/evalhub/single_tenancy/test_evalhub_single_tenancy_k8s.py::TestEvalHubSingleTenancyOperatorResources::test_deployment_created_and_ready[model_namespace0] - timeout_sampler.TimeoutExpiredError: Timed Out: 296.93537950515747 Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance Last exception: N/A. Elapsed time: 296.93537950515747 [0:04:56.935380] ERROR tests/ai_safety/evalhub/single_tenancy/test_evalhub_single_tenancy_k8s.py::TestEvalHubSingleTenancyOperatorResources::test_route_created[model_namespace0] - timeout_sampler.TimeoutExpiredError: Timed Out: 296.93537950515747 Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance Last exception: N/A. Elapsed time: 296.93537950515747 [0:04:56.935380] ==== 4 passed, 298 deselected, 417 warnings, 2 errors in 312.22s (0:05:12) =====