--- BVT pytest attempt 2026-06-12T14:00:51.769514+00:00 (/.local/bin/uv run pytest...) --- 2026-06-12T14:00:52.969955Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-06-12T14:00:52.980706Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.980904Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.981072Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.981580Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.981746Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.981889Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.982455Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.982589Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.982744Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.982889Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.983699Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.983845Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.984041Z [info ] [no key 0.00007s] ()  [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.984167Z [info ] [no key 0.00007s] ()  [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.984799Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.984947Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:00:52.987343Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-12T14:00:52.995899Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-06-12T14:00:53.074152Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-12T14:00:53.105425Z [info ] Running model registry tests against default namespace [conftest] 2026-06-12T14:00:53.105599Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] ============================= test session starts ============================== platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.14.5', 'Platform': 'Linux-5.14.0-427.115.1.el9_4.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.0.3', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.13.0', 'asyncio': '1.3.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.3.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.1.0'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.13.0, asyncio-1.3.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.3.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.1.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... collected 49 items / 42 deselected / 7 selected tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment]  -------------------------- test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment] -------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-12T14:00:53.329464Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() started [sqlalchemy.orm.mapper.Mapper] 2026-06-12T14:00:53.329666Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop id [sqlalchemy.orm.mapper.Mapper] 2026-06-12T14:00:53.329836Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop test_name [sqlalchemy.orm.mapper.Mapper] 2026-06-12T14:00:53.330690Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop start_time [sqlalchemy.orm.mapper.Mapper] 2026-06-12T14:00:53.330835Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() complete [sqlalchemy.orm.mapper.Mapper] 2026-06-12T14:00:53.330998Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:00:53.331128Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:00:53.331253Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:00:53.331372Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:00:53.331530Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment]', 1781272853) [sqlalchemy.engine.Engine] 2026-06-12T14:00:53.331673Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment]', 1781272853) [sqlalchemy.engine.Engine] 2026-06-12T14:00:53.332314Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:00:53.332445Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:00:53.336344Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-06-12T14:00:53.336499Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-06-12T14:00:53.336651Z [info ] Executing session fixture: admin_client [conftest] 2026-06-12T14:00:53.336800Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-12T14:00:53.343651Z [info ] Executing session fixture: dsc_resource [conftest] 2026-06-12T14:00:53.343797Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-06-12T14:00:53.445621Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-06-12T14:00:53.445800Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-06-12T14:00:53.446805Z [info ] Executing session fixture: os_path_environment [conftest] 2026-06-12T14:00:53.446968Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-06-12T14:00:53.447140Z [info ] Executing session fixture: bin_directory [conftest] 2026-06-12T14:00:53.448330Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-06-12T14:00:53.448504Z [info ] kind: ConsoleCLIDownload api version: console.openshift.io/v1 [ocp_resources.resource] 2026-06-12T14:00:53.485085Z [info ] All oc console cli download links: ['https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar'] [utilities.infra] name=utilities.infra 2026-06-12T14:00:53.485170Z [info ] Downloading archive using: url=https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar [utilities.infra] name=utilities.infra 2026-06-12T14:00:55.213393Z [info ] Extract the downloaded archive. [utilities.infra] name=utilities.infra 2026-06-12T14:00:55.393323Z [info ] Downloaded file: ['oc']  [utilities.infra] name=utilities.infra 2026-06-12T14:00:55.424649Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-06-12T14:00:55.420493Z [info ] OC binary path: /tmp/pytest-7TDypyqyxGJBADFrCJugFX/bin0/oc [tests.conftest] name=tests.conftest 2026-06-12T14:00:55.420603Z [info ] Adding /tmp/pytest-7TDypyqyxGJBADFrCJugFX/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-06-12T14:00:55.425151Z [info ] Executing session fixture: nodes [conftest] 2026-06-12T14:00:55.494780Z [info ] Executing session fixture: dsci_resource [conftest] 2026-06-12T14:00:55.495121Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-12T14:00:55.509381Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-06-12T14:00:55.509544Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-06-12T14:00:55.509701Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-06-12T14:00:55.508634Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-06-12T14:00:55.510106Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-06-12T14:00:55.626385Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-06-12T14:00:55.709028Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsci_status_ready Kwargs: {'dsci_resource': }) [timeout_sampler] 2026-06-12T14:00:55.708248Z [info ] Wait for DSCI default-dsci to be in Ready status. [utilities.infra] name=utilities.infra 2026-06-12T14:00:55.709369Z [info ] Get DSCInitialization default-dsci status [ocp_resources DSCInitialization] 2026-06-12T14:00:55.723026Z [info ] Elapsed time: 0.0001888275146484375 [0:00:00.000189] [timeout_sampler] 2026-06-12T14:00:55.723188Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': }) [timeout_sampler] 2026-06-12T14:00:55.722262Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-12T14:00:55.723409Z [info ] Get DataScienceCluster default-dsc status [ocp_resources DataScienceCluster] 2026-06-12T14:00:55.741422Z [info ] Elapsed time: 4.5299530029296875e-05 [0:00:00.000045] [timeout_sampler] 2026-06-12T14:00:55.741814Z [info ] Executing session fixture: autouse_fixtures [conftest] 2026-06-12T14:00:55.742132Z [info ] Executing session fixture: pytestconfig [conftest] 2026-06-12T14:00:55.742245Z [info ] Executing session fixture: teardown_resources [conftest] 2026-06-12T14:00:55.742357Z [info ] Executing session fixture: mlserver_runtime_image [conftest] 2026-06-12T14:00:55.742474Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-12T14:00:55.742587Z [info ] Create Namespace sklearn-modelcar [ocp_resources Namespace] 2026-06-12T14:00:55.742698Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'sklearn-modelcar'}, 'spec': {}} [ocp_resources Namespace] 2026-06-12T14:00:55.775618Z [info ] Wait for Namespace sklearn-modelcar status to be Active [ocp_resources Namespace] 2026-06-12T14:00:55.775990Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-12T14:00:55.802480Z [info ] Status of Namespace sklearn-modelcar is Active [ocp_resources Namespace] 2026-06-12T14:00:55.802776Z [info ] Elapsed time: 0.00010919570922851562 [0:00:00.000109] [timeout_sampler] 2026-06-12T14:00:55.803094Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-12T14:00:55.803209Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-12T14:00:55.860574Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-12T14:00:55.861051Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-12T14:00:55.861230Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'sklearn-modelcar'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:00:55.910976Z [info ] Executing class fixture: mlserver_model_car_inference_service [conftest] 2026-06-12T14:00:55.911265Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-12T14:00:55.911399Z [info ] Create InferenceService sklearn-modelcar [ocp_resources InferenceService] 2026-06-12T14:00:55.911521Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'sklearn-modelcar', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'sklearn-modelcar'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'sklearn'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55'}}}} [ocp_resources InferenceService] 2026-06-12T14:00:55.999441Z [info ] Wait for InferenceService/sklearn-modelcar's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-12T14:00:55.999724Z [info ] Wait until InferenceService sklearn-modelcar is created [ocp_resources InferenceService] 2026-06-12T14:00:55.999885Z [info ] Waiting for 899.9999704360962 seconds [0:14:59.999970], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-12T14:00:56.019216Z [info ] Elapsed time: 7.915496826171875e-05 [0:00:00.000079] [timeout_sampler] 2026-06-12T14:00:56.019399Z [info ] Waiting for 899.9799842834473 seconds [0:14:59.979984], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-12T14:01:13.330961Z [info ] Elapsed time: 17.29617476463318 [0:00:17.296175] [timeout_sampler] 2026-06-12T14:01:13.345718Z [info ] Waiting for 882.6531641483307 seconds [0:14:42.653164], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-12T14:01:13.360447Z [info ] Elapsed time: 0.00011301040649414062 [0:00:00.000113] [timeout_sampler] 2026-06-12T14:01:13.374336Z [info ] Executing function fixture: snapshot [conftest] 2026-06-12T14:01:13.375146Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-12T14:01:13Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED _ 1 of 7 completed, 1 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[xgboost-modelcar-RawDeployment]  TEST: TestMLServerModelCar.test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment] STATUS: PASSED ------------------------------------------------------- TEARDOWN -------------------------------------------------------  -------------------------- test_mlserver_model_car_inference[xgboost-modelcar-RawDeployment] -------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-12T14:01:13.669382Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:01:13.669534Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:01:13.669685Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:01:13.669817Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:01:13.670393Z [info ] [cached since 20.34s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[xgboost-modelcar-RawDeployment]', 1781272873) [sqlalchemy.engine.Engine] 2026-06-12T14:01:13.670573Z [info ] [cached since 20.34s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[xgboost-modelcar-RawDeployment]', 1781272873) [sqlalchemy.engine.Engine] 2026-06-12T14:01:13.671729Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:01:13.671873Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:01:13.672034Z [info ] Delete InferenceService sklearn-modelcar [ocp_resources InferenceService] 2026-06-12T14:01:13.705592Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'sklearn', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-12T14:00:55Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:00:55Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-12T14:00:56Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-12T14:01:12Z'}], 'name': 'sklearn-modelcar', 'namespace': 'sklearn-modelcar', 'resourceVersion': '178437144', 'uid': 'd9113715-3aa1-4ce7-9fab-cb6d2d86eeb1'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'sklearn'}, 'name': '', 'resources': {}, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55'}}}, 'status': {'address': {'url': 'http://sklearn-modelcar-predictor.sklearn-modelcar.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://sklearn-modelcar-predictor-sklearn-modelcar.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-12T14:00:56Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-12T14:01:12Z', 'message': 'ReplicaSet "sklearn-modelcar-predictor-d564bd6d5" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-12T14:01:12Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-12T14:00:56Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://sklearn-modelcar-predictor.sklearn-modelcar.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-12T14:01:13.725644Z [info ] Wait until InferenceService sklearn-modelcar is deleted [ocp_resources InferenceService] 2026-06-12T14:01:13.725764Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:01:14.756227Z [info ] Elapsed time: 1.0139992237091064 [0:00:01.013999] [timeout_sampler] 2026-06-12T14:01:14.756389Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-12T14:01:14.787517Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-12T14:00:55Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:00:55Z'}], 'name': 'mlserver-runtime', 'namespace': 'sklearn-modelcar', 'resourceVersion': '178436642', 'uid': '3c836008-9407-48df-aaae-cb1992daded1'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:01:14.803880Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-12T14:01:14.804113Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:01:14.819057Z [info ] Elapsed time: 3.600120544433594e-05 [0:00:00.000036] [timeout_sampler] 2026-06-12T14:01:14.889252Z [info ] Delete Namespace sklearn-modelcar [ocp_resources Namespace] 2026-06-12T14:01:14.921142Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'sklearn-modelcar', 'uid': 'ea1d8123-46f6-4bfc-be8b-9a55017cd278', 'resourceVersion': '178436635', 'creationTimestamp': '2026-06-12T14:00:55Z', 'labels': {'kubernetes.io/metadata.name': 'sklearn-modelcar', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c39,c34', 'openshift.io/sa.scc.supplemental-groups': '1001550000/10000', 'openshift.io/sa.scc.uid-range': '1001550000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-12T14:00:55Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:00:55Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:00:55Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-12T14:01:14.985864Z [info ] Wait until Namespace sklearn-modelcar is deleted [ocp_resources Namespace] 2026-06-12T14:01:14.986093Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:01:24.161852Z [info ] Elapsed time: 9.159501791000366 [0:00:09.159502] [timeout_sampler] 2026-06-12T14:01:24.162117Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-12T14:01:24.162235Z [info ] Create Namespace xgboost-modelcar [ocp_resources Namespace] 2026-06-12T14:01:24.162346Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'xgboost-modelcar'}, 'spec': {}} [ocp_resources Namespace] 2026-06-12T14:01:24.193793Z [info ] Wait for Namespace xgboost-modelcar status to be Active [ocp_resources Namespace] 2026-06-12T14:01:24.193976Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-12T14:01:24.211774Z [info ] Status of Namespace xgboost-modelcar is Active [ocp_resources Namespace] 2026-06-12T14:01:24.212014Z [info ] Elapsed time: 4.57763671875e-05 [0:00:00.000046] [timeout_sampler] 2026-06-12T14:01:24.212125Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-12T14:01:24.212246Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-12T14:01:24.259733Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-12T14:01:24.259891Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-12T14:01:24.260063Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'xgboost-modelcar'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:01:24.297876Z [info ] Executing class fixture: mlserver_model_car_inference_service [conftest] 2026-06-12T14:01:24.298087Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-12T14:01:24.298218Z [info ] Create InferenceService xgboost-modelcar [ocp_resources InferenceService] 2026-06-12T14:01:24.298341Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'xgboost-modelcar', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'xgboost-modelcar'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'xgboost'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-xgboost@sha256:5b6982bdc939b53a7a1210f56aa52bf7de0f0cbc693668db3fd1f496571bff29'}}}} [ocp_resources InferenceService] 2026-06-12T14:01:24.340304Z [info ] Wait for InferenceService/xgboost-modelcar's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-12T14:01:24.340461Z [info ] Wait until InferenceService xgboost-modelcar is created [ocp_resources InferenceService] 2026-06-12T14:01:24.340657Z [info ] Waiting for 899.9999918937683 seconds [0:14:59.999992], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-12T14:01:24.353265Z [info ] Elapsed time: 4.982948303222656e-05 [0:00:00.000050] [timeout_sampler] 2026-06-12T14:01:24.353415Z [info ] Waiting for 899.986973285675 seconds [0:14:59.986973], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-12T14:01:40.672293Z [info ] Elapsed time: 16.304540872573853 [0:00:16.304541] [timeout_sampler] 2026-06-12T14:01:40.697198Z [info ] Waiting for 883.6431746482849 seconds [0:14:43.643175], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-12T14:01:40.711889Z [info ] Elapsed time: 5.698204040527344e-05 [0:00:00.000057] [timeout_sampler] 2026-06-12T14:01:40.727059Z [info ] Executing function fixture: snapshot [conftest] 2026-06-12T14:01:40.727196Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-12T14:01:40Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED _ 2 of 7 completed, 2 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment]  TEST: TestMLServerModelCar.test_mlserver_model_car_inference[xgboost-modelcar-RawDeployment] STATUS: PASSED ------------------------------------------------------- TEARDOWN -------------------------------------------------------  -------------------------- test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment] -------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-12T14:01:41.001161Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:01:41.001321Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:01:41.001490Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:01:41.001633Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:01:41.002062Z [info ] [cached since 47.67s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment]', 1781272901) [sqlalchemy.engine.Engine] 2026-06-12T14:01:41.003071Z [info ] [cached since 47.67s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment]', 1781272901) [sqlalchemy.engine.Engine] 2026-06-12T14:01:41.003220Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:01:41.003332Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:01:41.003437Z [info ] Delete InferenceService xgboost-modelcar [ocp_resources InferenceService] 2026-06-12T14:01:41.050318Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'xgboost', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-12T14:01:24Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:01:24Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-12T14:01:24Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-12T14:01:39Z'}], 'name': 'xgboost-modelcar', 'namespace': 'xgboost-modelcar', 'resourceVersion': '178438021', 'uid': '13c4c589-a374-43fa-ab75-09a7dfc07c79'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'xgboost'}, 'name': '', 'resources': {}, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-xgboost@sha256:5b6982bdc939b53a7a1210f56aa52bf7de0f0cbc693668db3fd1f496571bff29'}}}, 'status': {'address': {'url': 'http://xgboost-modelcar-predictor.xgboost-modelcar.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://xgboost-modelcar-predictor-xgboost-modelcar.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-12T14:01:24Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-12T14:01:39Z', 'message': 'ReplicaSet "xgboost-modelcar-predictor-7d6cc48f8c" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-12T14:01:39Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-12T14:01:24Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://xgboost-modelcar-predictor.xgboost-modelcar.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-12T14:01:41.070721Z [info ] Wait until InferenceService xgboost-modelcar is deleted [ocp_resources InferenceService] 2026-06-12T14:01:41.070871Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:01:42.101619Z [info ] Elapsed time: 1.0141749382019043 [0:00:01.014175] [timeout_sampler] 2026-06-12T14:01:42.101797Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-12T14:01:42.133350Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-12T14:01:24Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:01:24Z'}], 'name': 'mlserver-runtime', 'namespace': 'xgboost-modelcar', 'resourceVersion': '178437531', 'uid': '6e6a87af-cb59-447b-a880-68be923e569d'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:01:42.155602Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-12T14:01:42.155777Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:01:42.170399Z [info ] Elapsed time: 7.843971252441406e-05 [0:00:00.000078] [timeout_sampler] 2026-06-12T14:01:42.224338Z [info ] Delete Namespace xgboost-modelcar [ocp_resources Namespace] 2026-06-12T14:01:42.256008Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'xgboost-modelcar', 'uid': '7ad1038a-85a7-433d-9542-289790a8486a', 'resourceVersion': '178437517', 'creationTimestamp': '2026-06-12T14:01:24Z', 'labels': {'kubernetes.io/metadata.name': 'xgboost-modelcar', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c40,c0', 'openshift.io/sa.scc.supplemental-groups': '1001560000/10000', 'openshift.io/sa.scc.uid-range': '1001560000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-12T14:01:24Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:01:24Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:01:24Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-12T14:01:42.274874Z [info ] Wait until Namespace xgboost-modelcar is deleted [ocp_resources Namespace] 2026-06-12T14:01:42.275111Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:01:50.434053Z [info ] Elapsed time: 8.132148742675781 [0:00:08.132149] [timeout_sampler] 2026-06-12T14:01:50.434310Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-12T14:01:50.434462Z [info ] Create Namespace lightgbm-modelcar [ocp_resources Namespace] 2026-06-12T14:01:50.434595Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'lightgbm-modelcar'}, 'spec': {}} [ocp_resources Namespace] 2026-06-12T14:01:50.479260Z [info ] Wait for Namespace lightgbm-modelcar status to be Active [ocp_resources Namespace] 2026-06-12T14:01:50.479399Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-12T14:01:50.496938Z [info ] Status of Namespace lightgbm-modelcar is Active [ocp_resources Namespace] 2026-06-12T14:01:50.497157Z [info ] Elapsed time: 4.887580871582031e-05 [0:00:00.000049] [timeout_sampler] 2026-06-12T14:01:50.497309Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-12T14:01:50.497445Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-12T14:01:50.565062Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-12T14:01:50.565209Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-12T14:01:50.565345Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'lightgbm-modelcar'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:01:50.658945Z [info ] Executing class fixture: mlserver_model_car_inference_service [conftest] 2026-06-12T14:01:50.659201Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-12T14:01:50.659347Z [info ] Create InferenceService lightgbm-modelcar [ocp_resources InferenceService] 2026-06-12T14:01:50.660010Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'lightgbm-modelcar', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'lightgbm-modelcar'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'lightgbm'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-lightgbm@sha256:77eb15a2eccefa3756faaf2ee4bc1e63990b746427d323957c461f33a4f1a6a3'}}}} [ocp_resources InferenceService] 2026-06-12T14:01:50.712709Z [info ] Wait for InferenceService/lightgbm-modelcar's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-12T14:01:50.712881Z [info ] Wait until InferenceService lightgbm-modelcar is created [ocp_resources InferenceService] 2026-06-12T14:01:50.713030Z [info ] Waiting for 899.9999969005585 seconds [0:14:59.999997], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-12T14:01:50.728164Z [info ] Elapsed time: 6.461143493652344e-05 [0:00:00.000065] [timeout_sampler] 2026-06-12T14:01:50.728333Z [info ] Waiting for 899.9843742847443 seconds [0:14:59.984374], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-12T14:02:06.010190Z [info ] Elapsed time: 15.266592025756836 [0:00:15.266592] [timeout_sampler] 2026-06-12T14:02:06.023503Z [info ] Waiting for 884.68887758255 seconds [0:14:44.688878], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-12T14:02:06.039869Z [info ] Elapsed time: 4.8160552978515625e-05 [0:00:00.000048] [timeout_sampler] 2026-06-12T14:02:06.055595Z [info ] Executing function fixture: snapshot [conftest] 2026-06-12T14:02:06.055760Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-12T14:02:06Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED TEST: TestMLServerModelCar.test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment] STATUS: PASSED _ 3 of 7 completed, 3 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment_text_type] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  --------------------- test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment_text_type] --------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-12T14:02:06.296695Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:02:06.296830Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:02:06.296974Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:02:06.297098Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:02:06.297242Z [info ] [cached since 72.97s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment_text_type]', 1781272926) [sqlalchemy.engine.Engine] 2026-06-12T14:02:06.297382Z [info ] [cached since 72.97s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment_text_type]', 1781272926) [sqlalchemy.engine.Engine] 2026-06-12T14:02:06.298527Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:02:06.298714Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:02:06.298903Z [info ] Delete InferenceService lightgbm-modelcar [ocp_resources InferenceService] 2026-06-12T14:02:06.331135Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'lightgbm', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-12T14:01:50Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:01:50Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-12T14:01:50Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-12T14:02:05Z'}], 'name': 'lightgbm-modelcar', 'namespace': 'lightgbm-modelcar', 'resourceVersion': '178438858', 'uid': 'bd559c72-af44-4407-9db7-e569ce9956e0'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'lightgbm'}, 'name': '', 'resources': {}, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-lightgbm@sha256:77eb15a2eccefa3756faaf2ee4bc1e63990b746427d323957c461f33a4f1a6a3'}}}, 'status': {'address': {'url': 'http://lightgbm-modelcar-predictor.lightgbm-modelcar.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://lightgbm-modelcar-predictor-lightgbm-modelcar.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-12T14:01:50Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-12T14:02:05Z', 'message': 'ReplicaSet "lightgbm-modelcar-predictor-5ccfc579b5" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-12T14:02:05Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-12T14:01:50Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://lightgbm-modelcar-predictor.lightgbm-modelcar.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-12T14:02:06.350385Z [info ] Wait until InferenceService lightgbm-modelcar is deleted [ocp_resources InferenceService] 2026-06-12T14:02:06.350546Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:02:07.379787Z [info ] Elapsed time: 1.014003038406372 [0:00:01.014003] [timeout_sampler] 2026-06-12T14:02:07.379979Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-12T14:02:07.422303Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-12T14:01:50Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:01:50Z'}], 'name': 'mlserver-runtime', 'namespace': 'lightgbm-modelcar', 'resourceVersion': '178438397', 'uid': 'c3f642d0-b32a-4e44-90fb-401f8dc57b0e'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:02:07.440322Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-12T14:02:07.440482Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:02:07.455796Z [info ] Elapsed time: 8.821487426757812e-05 [0:00:00.000088] [timeout_sampler] 2026-06-12T14:02:07.511517Z [info ] Delete Namespace lightgbm-modelcar [ocp_resources Namespace] 2026-06-12T14:02:07.539818Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'lightgbm-modelcar', 'uid': '754154bc-7527-4f22-bc63-bd866b46682b', 'resourceVersion': '178438380', 'creationTimestamp': '2026-06-12T14:01:50Z', 'labels': {'kubernetes.io/metadata.name': 'lightgbm-modelcar', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c40,c5', 'openshift.io/sa.scc.supplemental-groups': '1001570000/10000', 'openshift.io/sa.scc.uid-range': '1001570000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-12T14:01:50Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:01:50Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:01:50Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-12T14:02:07.560084Z [info ] Wait until Namespace lightgbm-modelcar is deleted [ocp_resources Namespace] 2026-06-12T14:02:07.560225Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:02:15.702864Z [info ] Elapsed time: 8.126031875610352 [0:00:08.126032] [timeout_sampler] 2026-06-12T14:02:15.703122Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-12T14:02:15.703271Z [info ] Create Namespace lightgbm-model-car-text-type [ocp_resources Namespace] 2026-06-12T14:02:15.703406Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'lightgbm-model-car-text-type'}, 'spec': {}} [ocp_resources Namespace] 2026-06-12T14:02:15.733584Z [info ] Wait for Namespace lightgbm-model-car-text-type status to be Active [ocp_resources Namespace] 2026-06-12T14:02:15.733715Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-12T14:02:15.751762Z [info ] Status of Namespace lightgbm-model-car-text-type is Active [ocp_resources Namespace] 2026-06-12T14:02:15.751908Z [info ] Elapsed time: 4.1484832763671875e-05 [0:00:00.000041] [timeout_sampler] 2026-06-12T14:02:15.752148Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-12T14:02:15.752306Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-12T14:02:15.793299Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-12T14:02:15.793506Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-12T14:02:15.793655Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'lightgbm-model-car-text-type'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:02:15.850427Z [info ] Executing class fixture: mlserver_model_car_inference_service [conftest] 2026-06-12T14:02:15.850639Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-12T14:02:15.850756Z [info ] Create InferenceService lightgbm-modelcar [ocp_resources InferenceService] 2026-06-12T14:02:15.850868Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'lightgbm-modelcar', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'lightgbm-model-car-text-type'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'lightgbm'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-lightgbm@sha256:77eb15a2eccefa3756faaf2ee4bc1e63990b746427d323957c461f33a4f1a6a3', 'env': [{'name': 'MLSERVER_MODEL_URI', 'value': '/mnt/models/model.txt'}]}}}} [ocp_resources InferenceService] 2026-06-12T14:02:15.911084Z [info ] Wait for InferenceService/lightgbm-modelcar's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-12T14:02:15.911247Z [info ] Wait until InferenceService lightgbm-modelcar is created [ocp_resources InferenceService] 2026-06-12T14:02:15.911388Z [info ] Waiting for 899.9999959468842 seconds [0:14:59.999996], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-12T14:02:15.935011Z [info ] Elapsed time: 8.296966552734375e-05 [0:00:00.000083] [timeout_sampler] 2026-06-12T14:02:15.935171Z [info ] Waiting for 899.9759504795074 seconds [0:14:59.975950], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-12T14:02:31.195349Z [info ] Elapsed time: 15.24447774887085 [0:00:15.244478] [timeout_sampler] 2026-06-12T14:02:31.209207Z [info ] Waiting for 884.7014291286469 seconds [0:14:44.701429], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-12T14:02:31.223164Z [info ] Elapsed time: 4.792213439941406e-05 [0:00:00.000048] [timeout_sampler] 2026-06-12T14:02:31.237822Z [info ] Executing function fixture: snapshot [conftest] 2026-06-12T14:02:31.238066Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-12T14:02:31Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED _ 4 of 7 completed, 4 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[onnx-modelcar-RawDeployment]  TEST: TestMLServerModelCar.test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment_text_type] STATUS: PASSED ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ---------------------------- test_mlserver_model_car_inference[onnx-modelcar-RawDeployment] ---------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-12T14:02:31.474781Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:02:31.475145Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:02:31.475304Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:02:31.476229Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:02:31.476406Z [info ] [cached since 98.15s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[onnx-modelcar-RawDeployment]', 1781272951) [sqlalchemy.engine.Engine] 2026-06-12T14:02:31.476628Z [info ] [cached since 98.15s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[onnx-modelcar-RawDeployment]', 1781272951) [sqlalchemy.engine.Engine] 2026-06-12T14:02:31.476761Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:02:31.476869Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:02:31.477002Z [info ] Delete InferenceService lightgbm-modelcar [ocp_resources InferenceService] 2026-06-12T14:02:31.508428Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'lightgbm', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-12T14:02:15Z', 'finalizers': ['odh.inferenceservice.finalizers', 'inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:env': {'.': {}, 'k:{"name":"MLSERVER_MODEL_URI"}': {'.': {}, 'f:name': {}, 'f:value': {}}}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:02:15Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-12T14:02:16Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-12T14:02:31Z'}], 'name': 'lightgbm-modelcar', 'namespace': 'lightgbm-model-car-text-type', 'resourceVersion': '178439688', 'uid': 'c2c9ff2a-34da-4739-b0b8-ae2e916c4b55'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'env': [{'name': 'MLSERVER_MODEL_URI', 'value': '/mnt/models/model.txt'}], 'modelFormat': {'name': 'lightgbm'}, 'name': '', 'resources': {}, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-lightgbm@sha256:77eb15a2eccefa3756faaf2ee4bc1e63990b746427d323957c461f33a4f1a6a3'}}}, 'status': {'address': {'url': 'http://lightgbm-modelcar-predictor.lightgbm-model-car-text-type.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://lightgbm-modelcar-predictor-lightgbm-model-car-text-type.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-12T14:02:16Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-12T14:02:31Z', 'message': 'ReplicaSet "lightgbm-modelcar-predictor-6f7f54d565" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-12T14:02:31Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-12T14:02:16Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://lightgbm-modelcar-predictor.lightgbm-model-car-text-type.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-12T14:02:31.526397Z [info ] Wait until InferenceService lightgbm-modelcar is deleted [ocp_resources InferenceService] 2026-06-12T14:02:31.526535Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:02:32.557703Z [info ] Elapsed time: 1.0136935710906982 [0:00:01.013694] [timeout_sampler] 2026-06-12T14:02:32.557857Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-12T14:02:32.589781Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-12T14:02:15Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:02:15Z'}], 'name': 'mlserver-runtime', 'namespace': 'lightgbm-model-car-text-type', 'resourceVersion': '178439205', 'uid': '6f040e82-195c-48e3-b30f-a43a83c88e30'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:02:32.606812Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-12T14:02:32.606966Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:02:32.622090Z [info ] Elapsed time: 0.00010943412780761719 [0:00:00.000109] [timeout_sampler] 2026-06-12T14:02:32.675099Z [info ] Delete Namespace lightgbm-model-car-text-type [ocp_resources Namespace] 2026-06-12T14:02:32.705463Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'lightgbm-model-car-text-type', 'uid': '1032cf5f-493e-4b36-88f1-57634017580d', 'resourceVersion': '178439199', 'creationTimestamp': '2026-06-12T14:02:15Z', 'labels': {'kubernetes.io/metadata.name': 'lightgbm-model-car-text-type', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c40,c10', 'openshift.io/sa.scc.supplemental-groups': '1001580000/10000', 'openshift.io/sa.scc.uid-range': '1001580000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-12T14:02:15Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:02:15Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:02:15Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-12T14:02:32.723394Z [info ] Wait until Namespace lightgbm-model-car-text-type is deleted [ocp_resources Namespace] 2026-06-12T14:02:32.723598Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:02:41.867275Z [info ] Elapsed time: 9.128124475479126 [0:00:09.128124] [timeout_sampler] 2026-06-12T14:02:41.867491Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-12T14:02:41.867607Z [info ] Create Namespace onnx-modelcar [ocp_resources Namespace] 2026-06-12T14:02:41.867720Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'onnx-modelcar'}, 'spec': {}} [ocp_resources Namespace] 2026-06-12T14:02:41.904560Z [info ] Wait for Namespace onnx-modelcar status to be Active [ocp_resources Namespace] 2026-06-12T14:02:41.904712Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-12T14:02:41.922224Z [info ] Status of Namespace onnx-modelcar is Active [ocp_resources Namespace] 2026-06-12T14:02:41.922383Z [info ] Elapsed time: 6.651878356933594e-05 [0:00:00.000067] [timeout_sampler] 2026-06-12T14:02:41.922496Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-12T14:02:41.922603Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-12T14:02:41.963048Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-12T14:02:41.963303Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-12T14:02:41.963469Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'onnx-modelcar'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:02:42.001976Z [info ] Executing class fixture: mlserver_model_car_inference_service [conftest] 2026-06-12T14:02:42.002198Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-12T14:02:42.002346Z [info ] Create InferenceService onnx-modelcar [ocp_resources InferenceService] 2026-06-12T14:02:42.002473Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'onnx-modelcar', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'onnx-modelcar'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'onnx'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-onnx@sha256:d0ad00fb6f2caa8f02a0250fc44a576771d0846b2ac8d164ec203b10ec5d604b'}}}} [ocp_resources InferenceService] 2026-06-12T14:02:42.089824Z [info ] Wait for InferenceService/onnx-modelcar's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-12T14:02:42.090064Z [info ] Wait until InferenceService onnx-modelcar is created [ocp_resources InferenceService] 2026-06-12T14:02:42.090276Z [info ] Waiting for 899.99999833107 seconds [0:14:59.999998], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-12T14:02:42.120386Z [info ] Elapsed time: 3.8623809814453125e-05 [0:00:00.000039] [timeout_sampler] 2026-06-12T14:02:42.120556Z [info ] Waiting for 899.9695687294006 seconds [0:14:59.969569], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-12T14:02:57.381197Z [info ] Elapsed time: 15.246907472610474 [0:00:15.246907] [timeout_sampler] 2026-06-12T14:02:57.395052Z [info ] Waiting for 884.6948773860931 seconds [0:14:44.694877], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-12T14:02:57.408859Z [info ] Elapsed time: 0.00010204315185546875 [0:00:00.000102] [timeout_sampler] 2026-06-12T14:02:57.424495Z [info ] Executing function fixture: snapshot [conftest] 2026-06-12T14:02:57.424653Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-12T14:02:57Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED TEST: TestMLServerModelCar.test_mlserver_model_car_inference[onnx-modelcar-RawDeployment] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-12T14:02:57.660784Z [info ] Delete InferenceService onnx-modelcar [ocp_resources InferenceService] 2026-06-12T14:02:57.692172Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'onnx', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-12T14:02:42Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:02:42Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-12T14:02:42Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-12T14:02:57Z'}], 'name': 'onnx-modelcar', 'namespace': 'onnx-modelcar', 'resourceVersion': '178440522', 'uid': 'e657297a-c4d8-44fa-bcb2-b3828d21801d'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'onnx'}, 'name': '', 'resources': {}, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-onnx@sha256:d0ad00fb6f2caa8f02a0250fc44a576771d0846b2ac8d164ec203b10ec5d604b'}}}, 'status': {'address': {'url': 'http://onnx-modelcar-predictor.onnx-modelcar.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://onnx-modelcar-predictor-onnx-modelcar.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-12T14:02:42Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-12T14:02:57Z', 'message': 'ReplicaSet "onnx-modelcar-predictor-f4c6ff9d" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-12T14:02:57Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-12T14:02:42Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://onnx-modelcar-predictor.onnx-modelcar.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-12T14:02:57.711968Z [info ] Wait until InferenceService onnx-modelcar is deleted [ocp_resources InferenceService] 2026-06-12T14:02:57.712177Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:02:58.741838Z [info ] Elapsed time: 1.0144398212432861 [0:00:01.014440] [timeout_sampler] 2026-06-12T14:02:58.742044Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-12T14:02:58.773337Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-12T14:02:41Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:02:41Z'}], 'name': 'mlserver-runtime', 'namespace': 'onnx-modelcar', 'resourceVersion': '178440044', 'uid': 'a1fd2d74-5841-4e63-a13b-763d08628b41'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:02:58.793166Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-12T14:02:58.793320Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:02:58.807454Z [info ] Elapsed time: 3.790855407714844e-05 [0:00:00.000038] [timeout_sampler] 2026-06-12T14:02:58.866630Z [info ] Delete Namespace onnx-modelcar [ocp_resources Namespace] 2026-06-12T14:02:58.898956Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'onnx-modelcar', 'uid': '354db89f-e96c-435d-b14d-5134d6a3e836', 'resourceVersion': '178440027', 'creationTimestamp': '2026-06-12T14:02:41Z', 'labels': {'kubernetes.io/metadata.name': 'onnx-modelcar', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c40,c15', 'openshift.io/sa.scc.supplemental-groups': '1001590000/10000', 'openshift.io/sa.scc.uid-range': '1001590000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-12T14:02:41Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:02:41Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:02:41Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-12T14:02:58.916764Z [info ] Wait until Namespace onnx-modelcar is deleted [ocp_resources Namespace] 2026-06-12T14:02:58.916912Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] _ 5 of 7 completed, 5 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/mlserver/s3/test_mlserver_s3.py::TestMLServerModels::test_mlserver_model_inference[sklearn-s3-RawDeployment] 2026-06-12T14:03:07.055276Z [info ] Elapsed time: 8.115288257598877 [0:00:08.115288] [timeout_sampler]  ------------------------------- test_mlserver_model_inference[sklearn-s3-RawDeployment] ------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-12T14:03:07.056566Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:03:07.056705Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:03:07.056832Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:03:07.057210Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:03:07.057362Z [info ] [cached since 133.7s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/s3/test_mlserver_s3.py::test_mlserver_model_inference[sklearn-s3-RawDeployment]', 1781272987) [sqlalchemy.engine.Engine] 2026-06-12T14:03:07.057495Z [info ] [cached since 133.7s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/s3/test_mlserver_s3.py::test_mlserver_model_inference[sklearn-s3-RawDeployment]', 1781272987) [sqlalchemy.engine.Engine] 2026-06-12T14:03:07.057614Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:03:07.057732Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:03:07.059727Z [info ] Executing session fixture: aws_access_key_id [conftest] 2026-06-12T14:03:07.059861Z [info ] Executing session fixture: aws_secret_access_key [conftest] 2026-06-12T14:03:07.060022Z [info ] Executing session fixture: ci_s3_bucket_endpoint [conftest] 2026-06-12T14:03:07.060143Z [info ] Executing session fixture: valid_aws_config [conftest] 2026-06-12T14:03:07.112711Z [info ] AWS credentials validated successfully via STS GetCallerIdentity [tests.conftest] name=tests.conftest 2026-06-12T14:03:07.115689Z [info ] Executing session fixture: models_s3_bucket_name [conftest] 2026-06-12T14:03:07.115989Z [info ] Executing session fixture: s3_models_storage_uri [conftest] 2026-06-12T14:03:07.116247Z [info ] Executing session fixture: models_s3_bucket_region [conftest] 2026-06-12T14:03:07.116359Z [info ] Executing session fixture: models_s3_bucket_endpoint [conftest] 2026-06-12T14:03:07.116465Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-12T14:03:07.116569Z [info ] Create Namespace sklearn-s3  [ocp_resources Namespace] 2026-06-12T14:03:07.116673Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'sklearn-s3'}, 'spec': {}} [ocp_resources Namespace] 2026-06-12T14:03:07.159999Z [info ] Wait for Namespace sklearn-s3 status to be Active [ocp_resources Namespace] 2026-06-12T14:03:07.160266Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-12T14:03:07.176660Z [info ] Status of Namespace sklearn-s3 is Active [ocp_resources Namespace] 2026-06-12T14:03:07.176813Z [info ] Elapsed time: 4.410743713378906e-05 [0:00:00.000044] [timeout_sampler] 2026-06-12T14:03:07.177394Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-12T14:03:07.177525Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-12T14:03:07.243757Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-12T14:03:07.244168Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-12T14:03:07.244329Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'sklearn-s3'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:03:07.280771Z [info ] Executing class fixture: kserve_s3_secret [conftest] 2026-06-12T14:03:07.280942Z [info ] Create Secret models-bucket-secret [ocp_resources Secret] 2026-06-12T14:03:07.281097Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'models-bucket-secret', 'annotations': {'serving.kserve.io/s3-endpoint': 's3.us-east-2.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-2', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-verifyssl': '0', 'serving.kserve.io/s3-usehttps': '1'}, 'namespace': 'sklearn-s3'}, 'stringData': '*******'} [ocp_resources Secret] 2026-06-12T14:03:07.316301Z [info ] Wait until Secret models-bucket-secret is created [ocp_resources Secret] 2026-06-12T14:03:07.316436Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-12T14:03:07.330394Z [info ] Elapsed time: 7.152557373046875e-05 [0:00:00.000072] [timeout_sampler] 2026-06-12T14:03:07.330527Z [info ] Executing class fixture: mlserver_model_service_account [conftest] 2026-06-12T14:03:07.330651Z [info ] Create ServiceAccount mlserver-models-bucket-sa [ocp_resources ServiceAccount] 2026-06-12T14:03:07.330777Z [info ] Posting {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'mlserver-models-bucket-sa', 'namespace': 'sklearn-s3'}, 'secrets': [{'name': 'models-bucket-secret'}]} [ocp_resources ServiceAccount] 2026-06-12T14:03:07.359900Z [info ] Executing class fixture: mlserver_inference_service [conftest] 2026-06-12T14:03:07.376326Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-12T14:03:07.376571Z [info ] Create InferenceService sklearn [ocp_resources InferenceService] 2026-06-12T14:03:07.376724Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'sklearn', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'sklearn-s3'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'sklearn'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 's3://ods-ci-wisdom/mlserver/model_repository/sklearn/', 'resources': {'requests': {'cpu': '1', 'memory': '2Gi'}, 'limits': {'cpu': '2', 'memory': '4Gi'}}}, 'minReplicas': 1, 'serviceAccountName': 'mlserver-models-bucket-sa'}}} [ocp_resources InferenceService] 2026-06-12T14:03:07.419559Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'mlserver-runtime'}) [timeout_sampler] 2026-06-12T14:03:07.418774Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-12T14:03:08.446491Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-12T14:03:08.493650Z [info ] Elapsed time: 1.027785301208496 [0:00:01.027785] [timeout_sampler] 2026-06-12T14:03:08.518368Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-06-12T14:03:08.519206Z [info ] Waiting for 899.9999969005585 seconds [0:14:59.999997], retry every 10 seconds. (Function: utilities.infra.get_pods_by_isvc_label Kwargs: {'client': , 'isvc': , 'runtime_name': 'mlserver-runtime'}) [timeout_sampler] 2026-06-12T14:03:18.665044Z [info ] Elapsed time: 10.061386108398438 [0:00:10.061386] [timeout_sampler] 2026-06-12T14:03:18.678698Z [info ] Waiting for 888.7405064105988 seconds [0:14:48.740506], retry every 5 seconds. (Function: ocp_resources.resource.get Kwargs: {'label_selector': 'serving.kserve.io/inferenceservice=sklearn', 'client': , 'namespace': 'sklearn-s3'}) [timeout_sampler] 2026-06-12T14:03:18.678860Z [info ] kind: Deployment api version: apps/v1 [ocp_resources.resource] 2026-06-12T14:03:18.692807Z [info ] Elapsed time: 5.817413330078125e-05 [0:00:00.000058] [timeout_sampler] 2026-06-12T14:03:18.692344Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-06-12T14:03:18.733555Z [info ] Waiting for 888.6854774951935 seconds [0:14:48.685477], retry every 5 seconds. (Function: utilities.infra..lambda: deployment.instance) [timeout_sampler] 2026-06-12T14:03:18.748583Z [info ] Elapsed time: 5.0067901611328125e-05 [0:00:00.000050] [timeout_sampler] 2026-06-12T14:03:18.748799Z [info ] Wait for Deployment sklearn-predictor to be deployed: True [ocp_resources Deployment] 2026-06-12T14:03:18.748967Z [info ] Waiting for 888.6704885959625 seconds [0:14:48.670489], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-06-12T14:03:18.762508Z [info ] Elapsed time: 3.0040740966796875e-05 [0:00:00.000030] [timeout_sampler] 2026-06-12T14:03:18.762713Z [info ] Waiting for 888.656534910202 seconds [0:14:48.656535], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-06-12T14:03:19.066470Z [info ] Elapsed time: 2.956390380859375e-05 [0:00:00.000030] [timeout_sampler] 2026-06-12T14:03:19.066704Z [info ] Wait for InferenceService/sklearn's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-12T14:03:19.066845Z [info ] Wait until InferenceService sklearn is created [ocp_resources InferenceService] 2026-06-12T14:03:19.067011Z [info ] Waiting for 888.3528075218201 seconds [0:14:48.352808], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-12T14:03:19.080575Z [info ] Elapsed time: 3.1948089599609375e-05 [0:00:00.000032] [timeout_sampler] 2026-06-12T14:03:19.080778Z [info ] Waiting for 888.3385117053986 seconds [0:14:48.338512], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-12T14:03:19.094774Z [info ] Elapsed time: 3.170967102050781e-05 [0:00:00.000032] [timeout_sampler] 2026-06-12T14:03:19.108387Z [info ] Waiting for 888.3108444213867 seconds [0:14:48.310844], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-12T14:03:19.122451Z [info ] Elapsed time: 0.00010752677917480469 [0:00:00.000108] [timeout_sampler] 2026-06-12T14:03:19.136428Z [info ] Executing function fixture: mlserver_pod_resource [conftest] 2026-06-12T14:03:19.165105Z [info ] Executing function fixture: snapshot [conftest] 2026-06-12T14:03:19.165683Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] 2026-06-12T14:03:19.166842Z [info ] Executing function fixture: model_format [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-12T14:03:19Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED TEST: TestMLServerModels.test_mlserver_model_inference[sklearn-s3-RawDeployment] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-12T14:03:19.334613Z [info ] Delete InferenceService sklearn [ocp_resources InferenceService] 2026-06-12T14:03:19.366854Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'sklearn', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-12T14:03:07Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:minReplicas': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:resources': {'.': {}, 'f:limits': {'.': {}, 'f:cpu': {}, 'f:memory': {}}, 'f:requests': {'.': {}, 'f:cpu': {}, 'f:memory': {}}}, 'f:runtime': {}, 'f:storageUri': {}}, 'f:serviceAccountName': {}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:03:07Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-12T14:03:07Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-12T14:03:18Z'}], 'name': 'sklearn', 'namespace': 'sklearn-s3', 'resourceVersion': '178441226', 'uid': 'ee95bbb4-de0a-4b68-ad19-0385a121fe71'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'minReplicas': 1, 'model': {'modelFormat': {'name': 'sklearn'}, 'name': '', 'resources': {'limits': {'cpu': '2', 'memory': '4Gi'}, 'requests': {'cpu': '1', 'memory': '2Gi'}}, 'runtime': 'mlserver-runtime', 'storageUri': 's3://ods-ci-wisdom/mlserver/model_repository/sklearn/'}, 'serviceAccountName': 'mlserver-models-bucket-sa'}}, 'status': {'address': {'url': 'http://sklearn-predictor.sklearn-s3.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://sklearn-predictor-sklearn-s3.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-12T14:03:07Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-12T14:03:18Z', 'message': 'ReplicaSet "sklearn-predictor-8594f6766d" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-12T14:03:18Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-12T14:03:07Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://sklearn-predictor.sklearn-s3.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-12T14:03:19.389445Z [info ] Wait until InferenceService sklearn is deleted [ocp_resources InferenceService] 2026-06-12T14:03:19.389768Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:03:20.422764Z [info ] Elapsed time: 1.0194146633148193 [0:00:01.019415] [timeout_sampler] 2026-06-12T14:03:20.423139Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-12T14:03:20.471677Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-12T14:03:07Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:03:07Z'}], 'name': 'mlserver-runtime', 'namespace': 'sklearn-s3', 'resourceVersion': '178440879', 'uid': 'b3419551-63b9-48e4-a00b-77fc63443feb'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:03:20.491840Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-12T14:03:20.492186Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:03:20.506467Z [info ] Elapsed time: 0.0001342296600341797 [0:00:00.000134] [timeout_sampler] 2026-06-12T14:03:20.506835Z [info ] Delete ServiceAccount mlserver-models-bucket-sa [ocp_resources ServiceAccount] 2026-06-12T14:03:20.553591Z [info ] Deleting {'kind': 'ServiceAccount', 'apiVersion': 'v1', 'metadata': {'name': 'mlserver-models-bucket-sa', 'namespace': 'sklearn-s3', 'uid': 'aa510f10-6784-4fb3-8cce-d53e5575dbeb', 'resourceVersion': '178440887', 'creationTimestamp': '2026-06-12T14:03:07Z', 'annotations': {'openshift.io/internal-registry-pull-secret-ref': 'mlserver-models-bucket-sa-dockercfg-qnbs5'}, 'managedFields': [{'manager': 'openshift.io/image-registry-pull-secrets_service-account-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-12T14:03:07Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:imagePullSecrets': {}, 'f:metadata': {'f:annotations': {'f:openshift.io/internal-registry-pull-secret-ref': {}}}, 'f:secrets': {'k:{"name":"mlserver-models-bucket-sa-dockercfg-qnbs5"}': {}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:03:07Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:secrets': {'.': {}, 'k:{"name":"models-bucket-secret"}': {}}}}]}, 'secrets': [{'name': 'models-bucket-secret'}, {'name': 'mlserver-models-bucket-sa-dockercfg-qnbs5'}], 'imagePullSecrets': [{'name': 'mlserver-models-bucket-sa-dockercfg-qnbs5'}]} [ocp_resources ServiceAccount] 2026-06-12T14:03:20.570732Z [info ] Wait until ServiceAccount mlserver-models-bucket-sa is deleted [ocp_resources ServiceAccount] 2026-06-12T14:03:20.571102Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:03:20.585120Z [info ] Elapsed time: 0.00010895729064941406 [0:00:00.000109] [timeout_sampler] 2026-06-12T14:03:20.585278Z [info ] Delete Secret models-bucket-secret [ocp_resources Secret] 2026-06-12T14:03:20.617717Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'models-bucket-secret', 'namespace': 'sklearn-s3', 'uid': 'bc53e079-2aa6-4b6d-9681-a0a75a2c5268', 'resourceVersion': '178440882', 'creationTimestamp': '2026-06-12T14:03:07Z', 'annotations': {'serving.kserve.io/s3-endpoint': 's3.us-east-2.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-2', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-usehttps': '1', 'serving.kserve.io/s3-verifyssl': '0'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:03:07Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:AWS_ACCESS_KEY_ID': {}, 'f:AWS_SECRET_ACCESS_KEY': {}}, 'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/s3-endpoint': {}, 'f:serving.kserve.io/s3-region': {}, 'f:serving.kserve.io/s3-useanoncredential': {}, 'f:serving.kserve.io/s3-usehttps': {}, 'f:serving.kserve.io/s3-verifyssl': {}}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-06-12T14:03:20.660964Z [info ] Wait until Secret models-bucket-secret is deleted [ocp_resources Secret] 2026-06-12T14:03:20.661172Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:03:20.689865Z [info ] Elapsed time: 9.5367431640625e-05 [0:00:00.000095] [timeout_sampler] 2026-06-12T14:03:20.869574Z [info ] Delete Namespace sklearn-s3  [ocp_resources Namespace] 2026-06-12T14:03:20.899293Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'sklearn-s3', 'uid': 'a307f073-3d8b-4c54-ab3e-6bad26cd28b8', 'resourceVersion': '178440865', 'creationTimestamp': '2026-06-12T14:03:07Z', 'labels': {'kubernetes.io/metadata.name': 'sklearn-s3', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c40,c20', 'openshift.io/sa.scc.supplemental-groups': '1001600000/10000', 'openshift.io/sa.scc.uid-range': '1001600000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-12T14:03:07Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:03:07Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:03:07Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-12T14:03:20.917911Z [info ] Wait until Namespace sklearn-s3 is deleted [ocp_resources Namespace] 2026-06-12T14:03:20.918208Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] _ 6 of 7 completed, 6 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::TestOpenVINOModels::test_openvino_model_inference[onnx-raw-rest-deployment] 2026-06-12T14:03:29.055216Z [info ] Elapsed time: 8.116556644439697 [0:00:08.116557] [timeout_sampler]  ------------------------------- test_openvino_model_inference[onnx-raw-rest-deployment] ------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-12T14:03:29.057798Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:03:29.058000Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:03:29.058163Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:03:29.058313Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-12T14:03:29.058503Z [info ] [cached since 155.7s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::test_openvino_model_inference[onnx-raw-rest-deployment]', 1781273009) [sqlalchemy.engine.Engine] 2026-06-12T14:03:29.058670Z [info ] [cached since 155.7s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::test_openvino_model_inference[onnx-raw-rest-deployment]', 1781273009) [sqlalchemy.engine.Engine] 2026-06-12T14:03:29.059518Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:03:29.059680Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-12T14:03:29.062417Z [info ] Executing session fixture: s3_models_storage_uri [conftest] 2026-06-12T14:03:29.062579Z [info ] Executing session fixture: gpu_count_on_cluster [conftest] 2026-06-12T14:03:29.162761Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-12T14:03:29.162948Z [info ] Create Namespace onnx-raw-rest [ocp_resources Namespace] 2026-06-12T14:03:29.163118Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'onnx-raw-rest'}, 'spec': {}} [ocp_resources Namespace] 2026-06-12T14:03:29.197609Z [info ] Wait for Namespace onnx-raw-rest status to be Active [ocp_resources Namespace] 2026-06-12T14:03:29.197823Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-12T14:03:29.219041Z [info ] Status of Namespace onnx-raw-rest is Active [ocp_resources Namespace] 2026-06-12T14:03:29.219354Z [info ] Elapsed time: 4.744529724121094e-05 [0:00:00.000047] [timeout_sampler] 2026-06-12T14:03:29.219634Z [info ] Executing class fixture: openvino_serving_runtime [conftest] 2026-06-12T14:03:29.219797Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-12T14:03:29.270236Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-12T14:03:29.270663Z [info ] Create ServingRuntime openvino-runtime [ocp_resources ServingRuntime] 2026-06-12T14:03:29.270856Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'openvino-runtime', 'namespace': 'onnx-raw-rest'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:1ab58519c50e2c3a9ebf0fee6d0708b1b5a0ae972aefcc722d87b2f62239a033', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:03:29.312252Z [info ] Executing class fixture: kserve_s3_secret [conftest] 2026-06-12T14:03:29.312646Z [info ] Create Secret models-bucket-secret [ocp_resources Secret] 2026-06-12T14:03:29.313317Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'models-bucket-secret', 'annotations': {'serving.kserve.io/s3-endpoint': 's3.us-east-2.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-2', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-verifyssl': '0', 'serving.kserve.io/s3-usehttps': '1'}, 'namespace': 'onnx-raw-rest'}, 'stringData': '*******'} [ocp_resources Secret] 2026-06-12T14:03:29.356558Z [info ] Wait until Secret models-bucket-secret is created [ocp_resources Secret] 2026-06-12T14:03:29.356795Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-12T14:03:29.375994Z [info ] Elapsed time: 4.315376281738281e-05 [0:00:00.000043] [timeout_sampler] 2026-06-12T14:03:29.376395Z [info ] Executing class fixture: openvino_model_service_account [conftest] 2026-06-12T14:03:29.376716Z [info ] Create ServiceAccount openvino-models-bucket-sa [ocp_resources ServiceAccount] 2026-06-12T14:03:29.376842Z [info ] Posting {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'openvino-models-bucket-sa', 'namespace': 'onnx-raw-rest'}, 'secrets': [{'name': 'models-bucket-secret'}]} [ocp_resources ServiceAccount] 2026-06-12T14:03:29.412552Z [info ] Executing class fixture: openvino_inference_service [conftest] 2026-06-12T14:03:29.447386Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-12T14:03:29.447753Z [info ] Create InferenceService onnx  [ocp_resources InferenceService] 2026-06-12T14:03:29.447886Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'onnx', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'onnx-raw-rest'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'openvino_ir'}, 'version': None, 'runtime': 'openvino-runtime', 'storageUri': 's3://ods-ci-wisdom/openvino/model_repository/onnx/', 'resources': {'requests': {'cpu': '1', 'memory': '2Gi'}, 'limits': {'cpu': '2', 'memory': '4Gi'}}}, 'minReplicas': 1, 'serviceAccountName': 'openvino-models-bucket-sa'}}} [ocp_resources InferenceService] 2026-06-12T14:03:29.502683Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'openvino-runtime'}) [timeout_sampler] 2026-06-12T14:03:29.501903Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-12T14:03:30.545004Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-12T14:03:30.602965Z [info ] Elapsed time: 1.0430965423583984 [0:00:01.043097] [timeout_sampler] 2026-06-12T14:03:30.636955Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-06-12T14:03:30.637764Z [info ] Waiting for 899.99999833107 seconds [0:14:59.999998], retry every 10 seconds. (Function: utilities.infra.get_pods_by_isvc_label Kwargs: {'client': , 'isvc': , 'runtime_name': 'openvino-runtime'}) [timeout_sampler] 2026-06-12T14:03:40.800609Z [info ] Elapsed time: 10.104259252548218 [0:00:10.104259] [timeout_sampler] 2026-06-12T14:03:40.814729Z [info ] Waiting for 888.6878898143768 seconds [0:14:48.687890], retry every 5 seconds. (Function: ocp_resources.resource.get Kwargs: {'label_selector': 'serving.kserve.io/inferenceservice=onnx', 'client': , 'namespace': 'onnx-raw-rest'}) [timeout_sampler] 2026-06-12T14:03:40.829236Z [info ] Elapsed time: 0.00010991096496582031 [0:00:00.000110] [timeout_sampler] 2026-06-12T14:03:40.828452Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-06-12T14:03:40.870395Z [info ] Waiting for 888.6321349143982 seconds [0:14:48.632135], retry every 5 seconds. (Function: utilities.infra..lambda: deployment.instance) [timeout_sampler] 2026-06-12T14:03:40.883390Z [info ] Elapsed time: 6.461143493652344e-05 [0:00:00.000065] [timeout_sampler] 2026-06-12T14:03:40.883614Z [info ] Wait for Deployment onnx-predictor to be deployed: True [ocp_resources Deployment] 2026-06-12T14:03:40.883741Z [info ] Waiting for 888.6191058158875 seconds [0:14:48.619106], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-06-12T14:03:40.897374Z [info ] Elapsed time: 3.1948089599609375e-05 [0:00:00.000032] [timeout_sampler] 2026-06-12T14:03:40.897605Z [info ] Waiting for 888.6051461696625 seconds [0:14:48.605146], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-06-12T14:03:40.912527Z [info ] Elapsed time: 3.075599670410156e-05 [0:00:00.000031] [timeout_sampler] 2026-06-12T14:03:40.912679Z [info ] Wait for InferenceService/onnx's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-12T14:03:40.912804Z [info ] Wait until InferenceService onnx is created [ocp_resources InferenceService] 2026-06-12T14:03:40.912938Z [info ] Waiting for 888.5900385379791 seconds [0:14:48.590039], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-12T14:03:40.926075Z [info ] Elapsed time: 2.86102294921875e-05 [0:00:00.000029] [timeout_sampler] 2026-06-12T14:03:40.926240Z [info ] Waiting for 888.5762438774109 seconds [0:14:48.576244], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-12T14:03:40.940803Z [info ] Elapsed time: 3.409385681152344e-05 [0:00:00.000034] [timeout_sampler] 2026-06-12T14:03:40.953880Z [info ] Waiting for 888.5483870506287 seconds [0:14:48.548387], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-12T14:03:40.968158Z [info ] Elapsed time: 6.031990051269531e-05 [0:00:00.000060] [timeout_sampler] 2026-06-12T14:03:40.982223Z [info ] Executing class fixture: protocol [conftest] 2026-06-12T14:03:40.982544Z [info ] Executing function fixture: openvino_pod_resource [conftest] 2026-06-12T14:03:41.010297Z [info ] Executing function fixture: snapshot [conftest] 2026-06-12T14:03:41.011272Z [info ] Executing function fixture: openvino_response_snapshot [conftest] 2026-06-12T14:03:41.012712Z [info ] Executing function fixture: model_format [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- FAILED TEST: TestOpenVINOModels.test_openvino_model_inference[onnx-raw-rest-deployment] STATUS: FAILED 2026-06-12T14:03:41.060582Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 179, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1720, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 166, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py", line 149, in test_openvino_model_inference validate_inference_request( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ pod_name=openvino_pod_resource.name, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... model_version=model_version, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/utils.py", line 113, in validate_inference_request response = run_openvino_inference( pod_name=pod_name, ...<2 lines>... model_version=model_version, ) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/utils.py", line 87, in run_openvino_inference raise ValueError(f"Invalid deployment_mode {deployment_mode}") ValueError: Invalid deployment_mode Standard [conftest] 2026-06-12T14:03:41.060724Z [info ] Must-gather collection is enabled for /home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::test_openvino_model_inference[onnx-raw-rest-deployment]. [conftest] 2026-06-12T14:03:41.060858Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:03:41.062397Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-12T14:03:41.062590Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-12T14:03:41.062747Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-12T14:03:41.063068Z [info ] [generated in 0.00023s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::test_openvino_model_inference[onnx-raw-rest-deployment]', 1, 0) [sqlalchemy.engine.Engine] 2026-06-12T14:03:41.065723Z [info ] [generated in 0.00023s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::test_openvino_model_inference[onnx-raw-rest-deployment]', 1, 0) [sqlalchemy.engine.Engine] 2026-06-12T14:03:41.065992Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-12T14:03:41.066318Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-12T14:03:41.066537Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-12T14:03:41.071637Z [info ] kind: Subscription api version: operators.coreos.com/v1alpha1 [ocp_resources.resource] 2026-06-12T14:03:41.152258Z [info ] Checking subscription rhods-operator [utilities.infra] name=utilities.infra 2026-06-12T14:03:41.168429Z [info ] Expected CSV: rhods-operator.3.4.0 [utilities.infra] name=utilities.infra 2026-06-12T14:03:41.171229Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-12T14:03:41.175980Z [info ] kind: ClusterServiceVersion api version: operators.coreos.com/v1alpha1 [ocp_resources.resource] 2026-06-12T14:03:41.325851Z [info ] Running oc adm must-gather --dest-dir=/home/odh/opendatahub-tests/results/must-gather-collected/model_serving/model_runtime/openvino/test_ovms_model_deployment/TestOpenVINOModels/test_openvino_model_inference_onnx-raw-rest-deployment_/pytest_exception_interact --since=300s --image=registry.redhat.io/rhoai/odh-must-gather-rhel9@sha256:3de5ff9eee7e9b8e6650f61dbae642db3d3ccb813afb804397dc60d18f150ea6 command [pyhelper_utils.shell] 2026-06-12T14:08:58.045867Z [info ] /home/odh/opendatahub-tests/results/must-gather-collected/model_serving/model_runtime/openvino/test_ovms_model_deployment/TestOpenVINOModels/test_openvino_model_inference_onnx-raw-rest-deployment_/pytest_exception_interact/mg-1781273009.zip is collected successfully [utilities.must_gather_collector] name=utilities.must_gather_collector ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-12T14:08:58.048043Z [info ] Delete InferenceService onnx  [ocp_resources InferenceService] 2026-06-12T14:08:58.159273Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'openvino_ir', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-12T14:03:29Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:minReplicas': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:resources': {'.': {}, 'f:limits': {'.': {}, 'f:cpu': {}, 'f:memory': {}}, 'f:requests': {'.': {}, 'f:cpu': {}, 'f:memory': {}}}, 'f:runtime': {}, 'f:storageUri': {}}, 'f:serviceAccountName': {}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:03:29Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-12T14:03:29Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-12T14:03:33Z'}], 'name': 'onnx', 'namespace': 'onnx-raw-rest', 'resourceVersion': '178441780', 'uid': '3d3f717b-0acc-409b-9f9b-639b7539a41a'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'minReplicas': 1, 'model': {'modelFormat': {'name': 'openvino_ir'}, 'name': '', 'resources': {'limits': {'cpu': '2', 'memory': '4Gi'}, 'requests': {'cpu': '1', 'memory': '2Gi'}}, 'runtime': 'openvino-runtime', 'storageUri': 's3://ods-ci-wisdom/openvino/model_repository/onnx/'}, 'serviceAccountName': 'openvino-models-bucket-sa'}}, 'status': {'address': {'url': 'http://onnx-predictor.onnx-raw-rest.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://onnx-predictor-onnx-raw-rest.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-12T14:03:29Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-12T14:03:33Z', 'message': 'ReplicaSet "onnx-predictor-5f854dc54c" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-12T14:03:33Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-12T14:03:29Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'openvino-runtime', 'url': 'http://onnx-predictor.onnx-raw-rest.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-12T14:08:58.191426Z [info ] Wait until InferenceService onnx is deleted [ocp_resources InferenceService] 2026-06-12T14:08:58.191582Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:08:59.237759Z [info ] Elapsed time: 1.0300931930541992 [0:00:01.030093] [timeout_sampler] 2026-06-12T14:08:59.237956Z [info ] Delete ServingRuntime openvino-runtime [ocp_resources ServingRuntime] 2026-06-12T14:08:59.270304Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'creationTimestamp': '2026-06-12T14:03:29Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/recommended-accelerators': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-12T14:03:29Z'}], 'name': 'openvino-runtime', 'namespace': 'onnx-raw-rest', 'resourceVersion': '178441608', 'uid': '4758315f-6688-40be-9c7e-59a284aa4302'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:1ab58519c50e2c3a9ebf0fee6d0708b1b5a0ae972aefcc722d87b2f62239a033', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-06-12T14:08:59.289257Z [info ] Wait until ServingRuntime openvino-runtime is deleted [ocp_resources ServingRuntime] 2026-06-12T14:08:59.289435Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:08:59.304855Z [info ] Elapsed time: 5.936622619628906e-05 [0:00:00.000059] [timeout_sampler] 2026-06-12T14:08:59.305118Z [info ] Delete ServiceAccount openvino-models-bucket-sa [ocp_resources ServiceAccount] 2026-06-12T14:08:59.335646Z [info ] Deleting {'kind': 'ServiceAccount', 'apiVersion': 'v1', 'metadata': {'name': 'openvino-models-bucket-sa', 'namespace': 'onnx-raw-rest', 'uid': '7faf6cac-746c-46ce-957a-7dc58f74ac1e', 'resourceVersion': '178441623', 'creationTimestamp': '2026-06-12T14:03:29Z', 'annotations': {'openshift.io/internal-registry-pull-secret-ref': 'openvino-models-bucket-sa-dockercfg-8x629'}, 'managedFields': [{'manager': 'openshift.io/image-registry-pull-secrets_service-account-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-12T14:03:29Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:imagePullSecrets': {}, 'f:metadata': {'f:annotations': {'f:openshift.io/internal-registry-pull-secret-ref': {}}}, 'f:secrets': {'k:{"name":"openvino-models-bucket-sa-dockercfg-8x629"}': {}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:03:29Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:secrets': {'.': {}, 'k:{"name":"models-bucket-secret"}': {}}}}]}, 'secrets': [{'name': 'models-bucket-secret'}, {'name': 'openvino-models-bucket-sa-dockercfg-8x629'}], 'imagePullSecrets': [{'name': 'openvino-models-bucket-sa-dockercfg-8x629'}]} [ocp_resources ServiceAccount] 2026-06-12T14:08:59.352837Z [info ] Wait until ServiceAccount openvino-models-bucket-sa is deleted [ocp_resources ServiceAccount] 2026-06-12T14:08:59.353046Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:08:59.368002Z [info ] Elapsed time: 4.5299530029296875e-05 [0:00:00.000045] [timeout_sampler] 2026-06-12T14:08:59.368238Z [info ] Delete Secret models-bucket-secret [ocp_resources Secret] 2026-06-12T14:08:59.401138Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'models-bucket-secret', 'namespace': 'onnx-raw-rest', 'uid': '44161425-521f-4afa-a01f-27f5a0b95748', 'resourceVersion': '178441612', 'creationTimestamp': '2026-06-12T14:03:29Z', 'annotations': {'serving.kserve.io/s3-endpoint': 's3.us-east-2.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-2', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-usehttps': '1', 'serving.kserve.io/s3-verifyssl': '0'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:03:29Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:AWS_ACCESS_KEY_ID': {}, 'f:AWS_SECRET_ACCESS_KEY': {}}, 'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/s3-endpoint': {}, 'f:serving.kserve.io/s3-region': {}, 'f:serving.kserve.io/s3-useanoncredential': {}, 'f:serving.kserve.io/s3-usehttps': {}, 'f:serving.kserve.io/s3-verifyssl': {}}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-06-12T14:08:59.417700Z [info ] Wait until Secret models-bucket-secret is deleted [ocp_resources Secret] 2026-06-12T14:08:59.417861Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:08:59.432470Z [info ] Elapsed time: 5.555152893066406e-05 [0:00:00.000056] [timeout_sampler] 2026-06-12T14:08:59.446486Z [info ] Delete Namespace onnx-raw-rest [ocp_resources Namespace] 2026-06-12T14:08:59.477681Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'onnx-raw-rest', 'uid': 'c74c8b6e-f354-48d0-9fc2-332a87cd9291', 'resourceVersion': '178441597', 'creationTimestamp': '2026-06-12T14:03:29Z', 'labels': {'kubernetes.io/metadata.name': 'onnx-raw-rest', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c40,c25', 'openshift.io/sa.scc.supplemental-groups': '1001610000/10000', 'openshift.io/sa.scc.uid-range': '1001610000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-12T14:03:29Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:03:29Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-12T14:03:29Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-12T14:08:59.498172Z [info ] Wait until Namespace onnx-raw-rest is deleted [ocp_resources Namespace] 2026-06-12T14:08:59.498389Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-12T14:09:12.737796Z [info ] Elapsed time: 13.21812105178833 [0:00:13.218121] [timeout_sampler] __ TestOpenVINOModels.test_openvino_model_inference[onnx-raw-rest-deployment] __ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 179, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1720, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 166, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py", line 149, in test_openvino_model_inference validate_inference_request( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ pod_name=openvino_pod_resource.name, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... model_version=model_version, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/utils.py", line 113, in validate_inference_request response = run_openvino_inference( pod_name=pod_name, ...<2 lines>... model_version=model_version, ) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/utils.py", line 87, in run_openvino_inference raise ValueError(f"Invalid deployment_mode {deployment_mode}") ValueError: Invalid deployment_mode Standard _ 7 of 7 completed, 6 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ ====== 1 failed, 6 passed, 42 deselected, 9 warnings in 499.63s (0:08:19) ====== =================================== FAILURES =================================== __ TestOpenVINOModels.test_openvino_model_inference[onnx-raw-rest-deployment] __ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 179, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1720, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 166, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py", line 149, in test_openvino_model_inference validate_inference_request( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ pod_name=openvino_pod_resource.name, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... model_version=model_version, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/utils.py", line 113, in validate_inference_request response = run_openvino_inference( pod_name=pod_name, ...<2 lines>... model_version=model_version, ) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/utils.py", line 87, in run_openvino_inference raise ValueError(f"Invalid deployment_mode {deployment_mode}") ValueError: Invalid deployment_mode Standard =============================== warnings summary =============================== tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment] tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[xgboost-modelcar-RawDeployment] tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment] tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment_text_type] tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[onnx-modelcar-RawDeployment] tests/model_serving/model_runtime/mlserver/s3/test_mlserver_s3.py::TestMLServerModels::test_mlserver_model_inference[sklearn-s3-RawDeployment] tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::TestOpenVINOModels::test_openvino_model_inference[onnx-raw-rest-deployment] tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::TestOpenVINOModels::test_openvino_model_inference[onnx-raw-rest-deployment] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::TestOpenVINOModels::test_openvino_model_inference[onnx-raw-rest-deployment] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py:1635: FutureWarning: 'client' arg will be mandatory in the next major release. `config_file` and `context` args will be removed. super().__init__( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ------------ generated xml file: /artifacts/model_runtime-smoke.xml ------------ =========================== short test summary info ============================ FAILED tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::TestOpenVINOModels::test_openvino_model_inference[onnx-raw-rest-deployment] - ValueError: Invalid deployment_mode Standard ====== 1 failed, 6 passed, 42 deselected, 9 warnings in 499.63s (0:08:19) ======