--- BVT pytest attempt 2026-06-02T07:24:11.980206+00:00 (/.local/bin/uv run pytest...) --- 2026-06-02T07:24:13.496678Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-06-02T07:24:13.506541Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.506816Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.507331Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.507536Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.507777Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.508579Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.509448Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.509744Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.509959Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.510186Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.510391Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.512759Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.513018Z [info ] [no key 0.00006s] ()  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.513228Z [info ] [no key 0.00006s] ()  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.517726Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.518116Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.518336Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-02T07:24:13.518537Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-09.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-09.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-09.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-06-02T07:24:13.652379Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-09.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( ============================= test session starts ============================== platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.14.5', 'Platform': 'Linux-5.14.0-427.115.1.el9_4.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.0.3', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.13.0', 'langsmith': '0.8.7', 'asyncio': '1.4.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.4.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.2.0'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.13.0, langsmith-0.8.7, asyncio-1.4.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.4.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.2.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... collected 87 items / 82 deselected / 5 selected tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment]  -------------------------- test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment] -------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-02T07:24:13.982060Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() started [sqlalchemy.orm.mapper.Mapper] 2026-06-02T07:24:13.982991Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop id [sqlalchemy.orm.mapper.Mapper] 2026-06-02T07:24:13.983710Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop test_name [sqlalchemy.orm.mapper.Mapper] 2026-06-02T07:24:13.983921Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop start_time [sqlalchemy.orm.mapper.Mapper] 2026-06-02T07:24:13.985357Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() complete [sqlalchemy.orm.mapper.Mapper] 2026-06-02T07:24:13.985603Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.985775Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.985949Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.987822Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.988136Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment]', 1780385053) [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.988362Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment]', 1780385053) [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.988561Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.989441Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-02T07:24:13.989616Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-06-02T07:24:13.989765Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-06-02T07:24:13.989908Z [info ] Executing session fixture: admin_client [conftest] 2026-06-02T07:24:13.996139Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-02T07:24:13.996435Z [info ] Executing session fixture: dsc_resource [conftest] 2026-06-02T07:24:13.996577Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-06-02T07:24:14.086468Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-06-02T07:24:14.086779Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-06-02T07:24:14.087732Z [info ] Executing session fixture: os_path_environment [conftest] 2026-06-02T07:24:14.087885Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-06-02T07:24:14.088063Z [info ] Executing session fixture: bin_directory [conftest] 2026-06-02T07:24:14.089324Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-06-02T07:24:14.089464Z [info ] kind: ConsoleCLIDownload api version: console.openshift.io/v1 [ocp_resources.resource] 2026-06-02T07:24:14.126949Z [info ] All oc console cli download links: ['https://downloads-openshift-console.apps.ods-qe-psi-09.osp.rh-ods.com/amd64/linux/oc.tar'] [utilities.infra] name=utilities.infra 2026-06-02T07:24:14.127093Z [info ] Downloading archive using: url=https://downloads-openshift-console.apps.ods-qe-psi-09.osp.rh-ods.com/amd64/linux/oc.tar [utilities.infra] name=utilities.infra 2026-06-02T07:24:16.651897Z [info ] Extract the downloaded archive. [utilities.infra] name=utilities.infra 2026-06-02T07:24:17.373213Z [info ] Downloaded file: ['oc']  [utilities.infra] name=utilities.infra 2026-06-02T07:24:17.407193Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-06-02T07:24:17.403154Z [info ] OC binary path: /tmp/pytest-SY9PgefuK4hpkhPALCqDLV/bin0/oc [tests.conftest] name=tests.conftest 2026-06-02T07:24:17.403263Z [info ] Adding /tmp/pytest-SY9PgefuK4hpkhPALCqDLV/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-06-02T07:24:17.407740Z [info ] Executing session fixture: nodes [conftest] 2026-06-02T07:24:17.497997Z [info ] Executing session fixture: dsci_resource [conftest] 2026-06-02T07:24:17.498182Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-02T07:24:17.516136Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-06-02T07:24:17.516396Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-06-02T07:24:17.516565Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-06-02T07:24:17.515182Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-06-02T07:24:17.516885Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-06-02T07:24:17.734126Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-06-02T07:24:17.874305Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsci_status_ready Kwargs: {'dsci_resource': }) [timeout_sampler] 2026-06-02T07:24:17.873409Z [info ] Wait for DSCI default-dsci to be in Ready status. [utilities.infra] name=utilities.infra 2026-06-02T07:24:17.874702Z [info ] Get DSCInitialization default-dsci status [ocp_resources DSCInitialization] 2026-06-02T07:24:17.891946Z [info ] Elapsed time: 0.00019502639770507812 [0:00:00.000195] [timeout_sampler] 2026-06-02T07:24:17.892122Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': }) [timeout_sampler] 2026-06-02T07:24:17.891217Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-02T07:24:17.892417Z [info ] Get DataScienceCluster default-dsc status [ocp_resources DataScienceCluster] 2026-06-02T07:24:17.917319Z [info ] Elapsed time: 4.029273986816406e-05 [0:00:00.000040] [timeout_sampler] 2026-06-02T07:24:17.917499Z [info ] Executing session fixture: autouse_fixtures [conftest] 2026-06-02T07:24:17.917668Z [info ] Executing session fixture: pytestconfig [conftest] 2026-06-02T07:24:17.917832Z [info ] Executing session fixture: teardown_resources [conftest] 2026-06-02T07:24:17.918022Z [info ] Executing session fixture: mlserver_runtime_image [conftest] 2026-06-02T07:24:17.918154Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-02T07:24:17.918292Z [info ] Create Namespace sklearn-modelcar [ocp_resources Namespace] 2026-06-02T07:24:17.918423Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'sklearn-modelcar'}, 'spec': {}} [ocp_resources Namespace] 2026-06-02T07:24:17.955245Z [info ] Wait for Namespace sklearn-modelcar status to be Active [ocp_resources Namespace] 2026-06-02T07:24:17.955446Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-02T07:24:17.983109Z [info ] Status of Namespace sklearn-modelcar is Active [ocp_resources Namespace] 2026-06-02T07:24:17.983274Z [info ] Elapsed time: 6.604194641113281e-05 [0:00:00.000066] [timeout_sampler] 2026-06-02T07:24:17.983426Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-02T07:24:17.983581Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-02T07:24:18.082130Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-02T07:24:18.082391Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-02T07:24:18.082595Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe - Tech Preview'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'sklearn-modelcar'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_NAME', 'value': '{{.Name}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:0c6c5ce6e4690aa7f60ca933046acedc650dbba07c6cde816454c5527e0cbd3c', 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}]}} [ocp_resources ServingRuntime] 2026-06-02T07:24:18.133671Z [info ] Executing class fixture: mlserver_model_car_inference_service [conftest] 2026-06-02T07:24:18.133957Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-02T07:24:18.134207Z [info ] Create InferenceService sklearn-modelcar [ocp_resources InferenceService] 2026-06-02T07:24:18.134400Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'sklearn-modelcar', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'sklearn-modelcar'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'sklearn'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55'}}}} [ocp_resources InferenceService] 2026-06-02T07:24:18.251425Z [info ] Wait for InferenceService/sklearn-modelcar's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-02T07:24:18.251623Z [info ] Wait until InferenceService sklearn-modelcar is created [ocp_resources InferenceService] 2026-06-02T07:24:18.251760Z [info ] Waiting for 899.9999732971191 seconds [0:14:59.999973], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-02T07:24:18.308307Z [info ] Elapsed time: 5.888938903808594e-05 [0:00:00.000059] [timeout_sampler] 2026-06-02T07:24:18.308510Z [info ] Waiting for 899.9431331157684 seconds [0:14:59.943133], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-02T07:24:24.430479Z [info ] Elapsed time: 6.10903000831604 [0:00:06.109030] [timeout_sampler] 2026-06-02T07:24:24.443005Z [info ] Waiting for 893.8081226348877 seconds [0:14:53.808123], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-02T07:24:24.455921Z [info ] Elapsed time: 5.984306335449219e-05 [0:00:00.000060] [timeout_sampler] 2026-06-02T07:24:24.469741Z [info ] Executing function fixture: snapshot [conftest] 2026-06-02T07:24:24.470934Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-02T07:24:24Z WARN hyper_rustls::config] native root CA certificate loading errors: [Error { context: "opening directory", kind: Io { inner: Os { code: 2, kind: NotFound, message: "No such file or directory" }, path: "/tekton-custom-certs:/etc/ssl/certs:/etc/pki/tls/certs" } }] FAILED TEST: TestMLServerModelCar.test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment] STATUS: FAILED 2026-06-02T07:24:24.567493Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 179, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1720, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 166, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py", line 133, in test_mlserver_model_car_inference validate_inference_request( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ pod_name=pod.name, ^^^^^^^^^^^^^^^^^^ ...<5 lines>... protocol=Protocols.REST, ^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/utils.py", line 114, in validate_inference_request response = run_mlserver_inference( pod_name=pod_name, ...<3 lines>... protocol=protocol, ) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/utils.py", line 83, in run_mlserver_inference with portforward.forward(pod_or_service=pod_name, namespace=isvc.namespace, from_port=port, to_port=port): ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.14/contextlib.py", line 141, in __enter__ return next(self.gen) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/portforward/__init__.py", line 89, in forward raise PortforwardError(err) from None portforward.PortforwardError: rustls tls error: no valid native root CA certificates found Caused by: 0: no valid native root CA certificates found 1: no native root CA certificates found (errors: [Error { context: "opening directory", kind: Io { inner: Os { code: 2, kind: NotFound, message: "No such file or directory" }, path: "/tekton-custom-certs:/etc/ssl/certs:/etc/pki/tls/certs" } }]) [conftest] 2026-06-02T07:24:24.571032Z [info ] Must-gather collection is enabled for /home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment]. [conftest] 2026-06-02T07:24:24.575467Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-02T07:24:24.575726Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-02T07:24:24.575892Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-02T07:24:24.576087Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-02T07:24:24.576347Z [info ] [generated in 0.00040s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment]', 1, 0) [sqlalchemy.engine.Engine] 2026-06-02T07:24:24.576507Z [info ] [generated in 0.00040s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment]', 1, 0) [sqlalchemy.engine.Engine] 2026-06-02T07:24:24.576639Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-02T07:24:24.576762Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-02T07:24:24.576895Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-02T07:24:24.577065Z [info ] kind: Subscription api version: operators.coreos.com/v1alpha1 [ocp_resources.resource] 2026-06-02T07:24:24.631615Z [info ] Checking subscription rhoai-operator-dev [utilities.infra] name=utilities.infra 2026-06-02T07:24:24.644626Z [info ] Expected CSV: rhods-operator.3.4.0-ea.1 [utilities.infra] name=utilities.infra 2026-06-02T07:24:24.647366Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-02T07:24:24.651721Z [info ] kind: ClusterServiceVersion api version: operators.coreos.com/v1alpha1 [ocp_resources.resource] 2026-06-02T07:24:24.852178Z [info ] Running oc adm must-gather --dest-dir=/home/odh/opendatahub-tests/results/must-gather-collected/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car/TestMLServerModelCar/test_mlserver_model_car_inference_sklearn-modelcar-RawDeployment_/pytest_exception_interact --since=300s --image=registry.redhat.io/rhoai/odh-must-gather-rhel9@sha256:7b394654b91821bd58cfe973c2d28f94b8ae323ed75e2e0de6fee1988b525690 command [pyhelper_utils.shell] /usr/lib64/python3.14/multiprocessing/resource_tracker.py:475: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown: {'/mp-b472qjwq', '/mp-tc6005ef', '/mp-df3w0fad'} warnings.warn(