--- BVT pytest attempt 2026-06-11T08:28:45.884691+00:00 (/.local/bin/uv run pytest...) --- 2026-06-11T08:28:47.047802Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-06-11T08:28:47.056886Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.057083Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.058183Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.058366Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.058517Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.059003Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.059165Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.059290Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.059464Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.059604Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.059730Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.059848Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.059982Z [info ] [no key 0.00005s] ()  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.060106Z [info ] [no key 0.00005s] ()  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.060232Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.060367Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.062606Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-11T08:28:47.069193Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-06-11T08:28:47.162800Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-11T08:28:47.189770Z [info ] Running model registry tests against default namespace [conftest] 2026-06-11T08:28:47.190197Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] ============================= test session starts ============================== platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.14.5', 'Platform': 'Linux-5.14.0-427.115.1.el9_4.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.0.3', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.13.0', 'asyncio': '1.3.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.3.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.1.0'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.13.0, asyncio-1.3.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.3.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.1.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... collected 87 items / 70 deselected / 17 selected tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::TestServingRuntimeImagesPerTemplate::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime0]  --------------- test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime0] --------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:28:47.492497Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() started [sqlalchemy.orm.mapper.Mapper] 2026-06-11T08:28:47.493165Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop id [sqlalchemy.orm.mapper.Mapper] 2026-06-11T08:28:47.493324Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop test_name [sqlalchemy.orm.mapper.Mapper] 2026-06-11T08:28:47.494181Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop start_time [sqlalchemy.orm.mapper.Mapper] 2026-06-11T08:28:47.494337Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() complete [sqlalchemy.orm.mapper.Mapper] 2026-06-11T08:28:47.494637Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.494769Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.494894Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.495017Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.495186Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime0]', 1781166527) [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.495331Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime0]', 1781166527) [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.496446Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.498946Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:28:47.499096Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-06-11T08:28:47.499236Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-06-11T08:28:47.499384Z [info ] Executing session fixture: admin_client [conftest] 2026-06-11T08:28:47.499521Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-11T08:28:47.503914Z [info ] Executing session fixture: dsc_resource [conftest] 2026-06-11T08:28:47.504048Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-06-11T08:28:47.578053Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-06-11T08:28:47.578333Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-06-11T08:28:47.579109Z [info ] Executing session fixture: os_path_environment [conftest] 2026-06-11T08:28:47.579250Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-06-11T08:28:47.579449Z [info ] Executing session fixture: bin_directory [conftest] 2026-06-11T08:28:47.580427Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-06-11T08:28:47.580629Z [info ] kind: ConsoleCLIDownload api version: console.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:28:47.612228Z [info ] All oc console cli download links: ['https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar'] [utilities.infra] name=utilities.infra 2026-06-11T08:28:47.612312Z [info ] Downloading archive using: url=https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar [utilities.infra] name=utilities.infra 2026-06-11T08:29:07.217451Z [info ] Extract the downloaded archive. [utilities.infra] name=utilities.infra 2026-06-11T08:29:07.353998Z [info ] Downloaded file: ['oc']  [utilities.infra] name=utilities.infra 2026-06-11T08:29:07.386146Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-06-11T08:29:07.381995Z [info ] OC binary path: /tmp/pytest-5wfkbJ2qUUMHDS9LeypewX/bin0/oc [tests.conftest] name=tests.conftest 2026-06-11T08:29:07.382091Z [info ] Adding /tmp/pytest-5wfkbJ2qUUMHDS9LeypewX/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-06-11T08:29:07.386642Z [info ] Executing session fixture: nodes [conftest] 2026-06-11T08:29:07.464259Z [info ] Executing session fixture: dsci_resource [conftest] 2026-06-11T08:29:07.464696Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-11T08:29:07.480293Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-06-11T08:29:07.480658Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-06-11T08:29:07.480886Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-06-11T08:29:07.479343Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-06-11T08:29:07.481139Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-06-11T08:29:07.566890Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-06-11T08:29:07.711995Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsci_status_ready Kwargs: {'dsci_resource': }) [timeout_sampler] 2026-06-11T08:29:07.711372Z [info ] Wait for DSCI default-dsci to be in Ready status. [utilities.infra] name=utilities.infra 2026-06-11T08:29:07.712253Z [info ] Get DSCInitialization default-dsci status [ocp_resources DSCInitialization] 2026-06-11T08:29:07.734521Z [info ] Elapsed time: 0.00011420249938964844 [0:00:00.000114] [timeout_sampler] 2026-06-11T08:29:07.734683Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': }) [timeout_sampler] 2026-06-11T08:29:07.733989Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T08:29:07.735065Z [info ] Get DataScienceCluster default-dsc status [ocp_resources DataScienceCluster] 2026-06-11T08:29:07.754775Z [info ] Elapsed time: 4.363059997558594e-05 [0:00:00.000044] [timeout_sampler] 2026-06-11T08:29:07.754944Z [info ] Executing session fixture: autouse_fixtures [conftest] 2026-06-11T08:29:07.755105Z [info ] Executing session fixture: related_images_refs [conftest] 2026-06-11T08:29:07.755270Z [info ] kind: ClusterServiceVersion api version: operators.coreos.com/v1alpha1 [ocp_resources.resource] 2026-06-11T08:29:07.846120Z [info ] Looking for rhods-operator.3.4.1 CSV in namespace redhat-ods-applications [utilities.operator_utils] name=utilities.operator_utils 2026-06-11T08:29:07.899530Z [info ] Found cluster service version: rhods-operator.3.4.1 [utilities.operator_utils] name=utilities.operator_utils 2026-06-11T08:29:07.928461Z [info ] Executing class fixture: serving_runtime_image_validation_namespace [conftest] 2026-06-11T08:29:07.928643Z [info ] Create Namespace runtime-verification [ocp_resources Namespace] 2026-06-11T08:29:07.928799Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'runtime-verification'}, 'spec': {}} [ocp_resources Namespace] 2026-06-11T08:29:07.973470Z [info ] Wait for Namespace runtime-verification status to be Active [ocp_resources Namespace] 2026-06-11T08:29:07.973648Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:07.987756Z [info ] Status of Namespace runtime-verification is Active [ocp_resources Namespace] 2026-06-11T08:29:07.987916Z [info ] Elapsed time: 4.6253204345703125e-05 [0:00:00.000046] [timeout_sampler] 2026-06-11T08:29:07.988069Z [info ] Executing function fixture: serving_runtime_pods_for_runtime [conftest] 2026-06-11T08:29:07.988222Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:29:08.033619Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:29:08.033782Z [info ] Create ServingRuntime odh-ovms-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:08.033985Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'odh-ovms-image-runtime', 'namespace': 'runtime-verification'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:a7a965959765c52a705ac632d19d823e4ad843ad3657e395d708da7e5401a29f', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:08.098556Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:29:08.098717Z [info ] Create InferenceService odh-ovms-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:08.098868Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'odh-ovms-image-isvc', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'runtime-verification'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'openvino_ir'}, 'version': None, 'runtime': 'odh-ovms-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}} [ocp_resources InferenceService] 2026-06-11T08:29:08.481269Z [info ] Waiting for 300 seconds [0:05:00], retry every 5 seconds. (Function: timeout_sampler.wrapper Kwargs: {'client': , 'isvc': , 'runtime_name': 'odh-ovms-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:08.481479Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'odh-ovms-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:08.480639Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:08.601719Z [info ] Elapsed time: 5.030632019042969e-05 [0:00:00.000050] [timeout_sampler] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-06-11T08:29:08.601142Z [info ] Validating odh-ovms-image-isvc-predictor-5d9d6447fc-j9gzv in runtime-verification [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images PASSED2026-06-11T08:29:08.683919Z [info ] odh_ovms_image : passed  [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images  TEST: TestServingRuntimeImagesPerTemplate.test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime0] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-11T08:29:08.689373Z [info ] Elapsed time: 6.914138793945312e-05 [0:00:00.000069] [timeout_sampler] 2026-06-11T08:29:08.689543Z [info ] Delete InferenceService odh-ovms-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:08.721320Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'openvino_ir', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:29:08Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:08Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:29:08Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:29:08Z'}], 'name': 'odh-ovms-image-isvc', 'namespace': 'runtime-verification', 'resourceVersion': '175445909', 'uid': 'c4c90070-e4c0-4106-b3d4-664e0ee2a84c'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'openvino_ir'}, 'name': '', 'resources': {}, 'runtime': 'odh-ovms-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}, 'status': {'address': {'url': 'http://odh-ovms-image-isvc-predictor.runtime-verification.svc.cluster.local:8080'}, 'components': {'predictor': {}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:29:08Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:29:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:29:08Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:29:08Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0}, 'states': {'activeModelState': '', 'targetModelState': 'Pending'}, 'transitionStatus': 'InProgress'}, 'observedGeneration': 1, 'servingRuntimeName': 'odh-ovms-image-runtime', 'url': 'http://odh-ovms-image-isvc-predictor.runtime-verification.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:29:08.740174Z [info ] Wait until InferenceService odh-ovms-image-isvc is deleted [ocp_resources InferenceService] 2026-06-11T08:29:08.740414Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:09.815990Z [info ] Elapsed time: 1.0135118961334229 [0:00:01.013512] [timeout_sampler] 2026-06-11T08:29:09.816181Z [info ] Delete ServingRuntime odh-ovms-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:09.846920Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'creationTimestamp': '2026-06-11T08:29:08Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/recommended-accelerators': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:08Z'}], 'name': 'odh-ovms-image-runtime', 'namespace': 'runtime-verification', 'resourceVersion': '175445837', 'uid': 'eeef0f55-3c00-41dd-a79a-1552ab2a19d5'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:a7a965959765c52a705ac632d19d823e4ad843ad3657e395d708da7e5401a29f', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:09.906214Z [info ] Wait until ServingRuntime odh-ovms-image-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:29:09.906460Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] _ 1 of 17 completed, 1 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::TestServingRuntimeImagesPerTemplate::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime1] 2026-06-11T08:29:09.926020Z [info ] Elapsed time: 7.605552673339844e-05 [0:00:00.000076] [timeout_sampler]  --------------- test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime1] --------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:29:09.928032Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:09.928148Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:09.928281Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:09.928430Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:09.928579Z [info ] [cached since 22.44s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime1]', 1781166549) [sqlalchemy.engine.Engine] 2026-06-11T08:29:09.928712Z [info ] [cached since 22.44s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime1]', 1781166549) [sqlalchemy.engine.Engine] 2026-06-11T08:29:09.929222Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:09.929340Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:09.930727Z [info ] Executing function fixture: serving_runtime_pods_for_runtime [conftest] 2026-06-11T08:29:09.930847Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:29:09.970677Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:29:09.970879Z [info ] Create ServingRuntime odh-vllm-cpu-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:09.971005Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': 'v0.18.0', 'openshift.io/display-name': 'vLLM CPU (x86) ServingRuntime for KServe'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'odh-vllm-cpu-image-runtime', 'namespace': 'runtime-verification'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'vllm', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8080'}, 'containers': [{'args': ['--port=8080', '--model=/mnt/models', '--served-model-name={{.Name}}'], 'command': ['python', '-m', 'vllm.entrypoints.openai.api_server'], 'env': [{'name': 'HF_HOME', 'value': '/tmp/hf_home'}], 'image': 'registry.redhat.io/rhaii/vllm-cpu-rhel9@sha256:c60b2eb7d53a44e11d24c8657d92562c0479e048a940d601b8d5e4d46c887e13', 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}]}], 'multiModel': False, 'supportedModelFormats': [{'autoSelect': True, 'name': 'vLLM'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:10.338889Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:29:10.339035Z [info ] Create InferenceService odh-vllm-cpu-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:10.339161Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'odh-vllm-cpu-image-isvc', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'runtime-verification'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'vLLM'}, 'version': None, 'runtime': 'odh-vllm-cpu-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}} [ocp_resources InferenceService] 2026-06-11T08:29:10.398165Z [info ] Waiting for 300 seconds [0:05:00], retry every 5 seconds. (Function: timeout_sampler.wrapper Kwargs: {'client': , 'isvc': , 'runtime_name': 'odh-vllm-cpu-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:10.398321Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'odh-vllm-cpu-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:10.397330Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:11.435227Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:11.465616Z [info ] Elapsed time: 1.0379235744476318 [0:00:01.037924] [timeout_sampler] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-06-11T08:29:11.464922Z [info ] Validating odh-vllm-cpu-image-isvc-predictor-7574fcb9f8-wnp5n in runtime-verification [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images PASSED2026-06-11T08:29:11.506755Z [info ] odh_vllm_cpu_image : passed  [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images  TEST: TestServingRuntimeImagesPerTemplate.test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime1] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-11T08:29:11.508411Z [info ] Elapsed time: 0.00010395050048828125 [0:00:00.000104] [timeout_sampler] 2026-06-11T08:29:11.508580Z [info ] Delete InferenceService odh-vllm-cpu-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:11.539021Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'vLLM', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:29:10Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:10Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:29:10Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:29:10Z'}], 'name': 'odh-vllm-cpu-image-isvc', 'namespace': 'runtime-verification', 'resourceVersion': '175446099', 'uid': '55a1e951-3f01-4a48-8aa9-06fd761b09a5'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'vLLM'}, 'name': '', 'resources': {}, 'runtime': 'odh-vllm-cpu-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}, 'status': {'address': {'url': 'http://odh-vllm-cpu-image-isvc-predictor.runtime-verification.svc.cluster.local:8080'}, 'components': {'predictor': {}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:29:10Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:29:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:29:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:29:10Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0}, 'states': {'activeModelState': '', 'targetModelState': 'Pending'}, 'transitionStatus': 'InProgress'}, 'observedGeneration': 1, 'servingRuntimeName': 'odh-vllm-cpu-image-runtime', 'url': 'http://odh-vllm-cpu-image-isvc-predictor.runtime-verification.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:29:11.565666Z [info ] Wait until InferenceService odh-vllm-cpu-image-isvc is deleted [ocp_resources InferenceService] 2026-06-11T08:29:11.565826Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:12.593873Z [info ] Elapsed time: 1.0134620666503906 [0:00:01.013462] [timeout_sampler] 2026-06-11T08:29:12.594017Z [info ] Delete ServingRuntime odh-vllm-cpu-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:12.623561Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': 'v0.18.0', 'openshift.io/display-name': 'vLLM CPU (x86) ServingRuntime for KServe'}, 'creationTimestamp': '2026-06-11T08:29:10Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:10Z'}], 'name': 'odh-vllm-cpu-image-runtime', 'namespace': 'runtime-verification', 'resourceVersion': '175446027', 'uid': 'b8033fd2-43b8-4327-81ae-f6e64d2a34c9'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'vllm', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8080'}, 'containers': [{'args': ['--port=8080', '--model=/mnt/models', '--served-model-name={{.Name}}'], 'command': ['python', '-m', 'vllm.entrypoints.openai.api_server'], 'env': [{'name': 'HF_HOME', 'value': '/tmp/hf_home'}], 'image': 'registry.redhat.io/rhaii/vllm-cpu-rhel9@sha256:c60b2eb7d53a44e11d24c8657d92562c0479e048a940d601b8d5e4d46c887e13', 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}]}], 'multiModel': False, 'supportedModelFormats': [{'autoSelect': True, 'name': 'vLLM'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:12.640286Z [info ] Wait until ServingRuntime odh-vllm-cpu-image-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:29:12.640531Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] _ 2 of 17 completed, 2 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::TestServingRuntimeImagesPerTemplate::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime2] 2026-06-11T08:29:12.654873Z [info ] Elapsed time: 8.392333984375e-05 [0:00:00.000084] [timeout_sampler]  --------------- test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime2] --------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:29:12.656024Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:12.656151Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:12.656267Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:12.656423Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:12.656772Z [info ] [cached since 25.17s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime2]', 1781166552) [sqlalchemy.engine.Engine] 2026-06-11T08:29:12.656909Z [info ] [cached since 25.17s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime2]', 1781166552) [sqlalchemy.engine.Engine] 2026-06-11T08:29:12.657024Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:12.657136Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:12.658214Z [info ] Executing function fixture: serving_runtime_pods_for_runtime [conftest] 2026-06-11T08:29:12.658337Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:29:12.764068Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:29:12.764707Z [info ] Create ServingRuntime odh-vllm-gaudi-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:12.764878Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["habana.ai/gaudi"]', 'opendatahub.io/runtime-version': 'v0.17.1', 'openshift.io/display-name': 'vLLM Intel Gaudi Accelerator ServingRuntime for KServe'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'odh-vllm-gaudi-image-runtime', 'namespace': 'runtime-verification'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'vllm', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8080'}, 'builtInAdapter': {'modelLoadingTimeoutMillis': 90000}, 'containers': [{'args': ['--port=8080', '--model=/mnt/models', '--served-model-name={{.Name}}'], 'command': ['python', '-m', 'vllm.entrypoints.openai.api_server'], 'env': [{'name': 'HF_HOME', 'value': '/tmp/hf_home'}], 'image': 'registry.redhat.io/rhaii/vllm-gaudi-rhel9@sha256:71008b2151586551ec0969ffc5b175f726ed6f6bebee29cdc289549e216609bc', 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}]}], 'multiModel': False, 'supportedModelFormats': [{'autoSelect': False, 'name': 'vLLM'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:12.820213Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:29:12.820400Z [info ] Create InferenceService odh-vllm-gaudi-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:12.820554Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'odh-vllm-gaudi-image-isvc', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'runtime-verification'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'vLLM'}, 'version': None, 'runtime': 'odh-vllm-gaudi-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}} [ocp_resources InferenceService] 2026-06-11T08:29:12.891428Z [info ] Waiting for 300 seconds [0:05:00], retry every 5 seconds. (Function: timeout_sampler.wrapper Kwargs: {'client': , 'isvc': , 'runtime_name': 'odh-vllm-gaudi-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:12.891582Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'odh-vllm-gaudi-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:12.890898Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:13.926811Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:13.967670Z [info ] Elapsed time: 1.0358920097351074 [0:00:01.035892] [timeout_sampler] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-06-11T08:29:13.966975Z [info ] Validating odh-vllm-gaudi-image-isvc-predictor-7698fcd699-98mhm in runtime-verification [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images PASSED2026-06-11T08:29:14.045477Z [info ] odh_vllm_gaudi_image : passed  [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images  TEST: TestServingRuntimeImagesPerTemplate.test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime2] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-11T08:29:14.046975Z [info ] Elapsed time: 6.198883056640625e-05 [0:00:00.000062] [timeout_sampler] 2026-06-11T08:29:14.047085Z [info ] Delete InferenceService odh-vllm-gaudi-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:14.128756Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'vLLM', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:29:12Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:12Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:29:13Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:29:13Z'}], 'name': 'odh-vllm-gaudi-image-isvc', 'namespace': 'runtime-verification', 'resourceVersion': '175446265', 'uid': '0940219b-912a-4de4-a9d4-8b39bb5e958b'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'vLLM'}, 'name': '', 'resources': {}, 'runtime': 'odh-vllm-gaudi-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}, 'status': {'address': {'url': 'http://odh-vllm-gaudi-image-isvc-predictor.runtime-verification.svc.cluster.local:8080'}, 'components': {'predictor': {}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:29:12Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:29:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:29:13Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:29:12Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0}, 'states': {'activeModelState': '', 'targetModelState': 'Pending'}, 'transitionStatus': 'InProgress'}, 'observedGeneration': 1, 'servingRuntimeName': 'odh-vllm-gaudi-image-runtime', 'url': 'http://odh-vllm-gaudi-image-isvc-predictor.runtime-verification.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:29:14.164629Z [info ] Wait until InferenceService odh-vllm-gaudi-image-isvc is deleted [ocp_resources InferenceService] 2026-06-11T08:29:14.164789Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:15.200843Z [info ] Elapsed time: 1.0188798904418945 [0:00:01.018880] [timeout_sampler] 2026-06-11T08:29:15.201082Z [info ] Delete ServingRuntime odh-vllm-gaudi-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:15.233036Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["habana.ai/gaudi"]', 'opendatahub.io/runtime-version': 'v0.17.1', 'openshift.io/display-name': 'vLLM Intel Gaudi Accelerator ServingRuntime for KServe'}, 'creationTimestamp': '2026-06-11T08:29:12Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/recommended-accelerators': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:builtInAdapter': {'.': {}, 'f:modelLoadingTimeoutMillis': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:12Z'}], 'name': 'odh-vllm-gaudi-image-runtime', 'namespace': 'runtime-verification', 'resourceVersion': '175446216', 'uid': 'c7b11279-286b-4f85-8a17-9df9d884eb63'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'vllm', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8080'}, 'builtInAdapter': {'modelLoadingTimeoutMillis': 90000}, 'containers': [{'args': ['--port=8080', '--model=/mnt/models', '--served-model-name={{.Name}}'], 'command': ['python', '-m', 'vllm.entrypoints.openai.api_server'], 'env': [{'name': 'HF_HOME', 'value': '/tmp/hf_home'}], 'image': 'registry.redhat.io/rhaii/vllm-gaudi-rhel9@sha256:71008b2151586551ec0969ffc5b175f726ed6f6bebee29cdc289549e216609bc', 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}]}], 'multiModel': False, 'supportedModelFormats': [{'autoSelect': False, 'name': 'vLLM'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:15.281225Z [info ] Wait until ServingRuntime odh-vllm-gaudi-image-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:29:15.281425Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] _ 3 of 17 completed, 3 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::TestServingRuntimeImagesPerTemplate::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime3] 2026-06-11T08:29:15.299270Z [info ] Elapsed time: 9.036064147949219e-05 [0:00:00.000090] [timeout_sampler]  --------------- test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime3] --------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:29:15.300536Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:15.300697Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:15.300847Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:15.300988Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:15.301653Z [info ] [cached since 27.81s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime3]', 1781166555) [sqlalchemy.engine.Engine] 2026-06-11T08:29:15.301822Z [info ] [cached since 27.81s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime3]', 1781166555) [sqlalchemy.engine.Engine] 2026-06-11T08:29:15.302480Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:15.302635Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:15.303944Z [info ] Executing function fixture: serving_runtime_pods_for_runtime [conftest] 2026-06-11T08:29:15.304099Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:29:15.341471Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:29:15.341669Z [info ] Create ServingRuntime odh-mlserver-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:15.341919Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'odh-mlserver-image-runtime', 'namespace': 'runtime-verification'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:15.398745Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:29:15.398941Z [info ] Create InferenceService odh-mlserver-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:15.399102Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'odh-mlserver-image-isvc', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'runtime-verification'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'sklearn'}, 'version': None, 'runtime': 'odh-mlserver-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}} [ocp_resources InferenceService] 2026-06-11T08:29:15.492604Z [info ] Waiting for 300 seconds [0:05:00], retry every 5 seconds. (Function: timeout_sampler.wrapper Kwargs: {'client': , 'isvc': , 'runtime_name': 'odh-mlserver-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:15.492772Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'odh-mlserver-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:15.491815Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:16.533487Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:16.563991Z [info ] Elapsed time: 1.0416970252990723 [0:00:01.041697] [timeout_sampler] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-06-11T08:29:16.563131Z [info ] Validating odh-mlserver-image-isvc-predictor-76df68c45c-dwztj in runtime-verification [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images PASSED2026-06-11T08:29:16.610651Z [info ] odh_mlserver_image : passed  [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images  TEST: TestServingRuntimeImagesPerTemplate.test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime3] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-11T08:29:16.612494Z [info ] Elapsed time: 0.00011277198791503906 [0:00:00.000113] [timeout_sampler] 2026-06-11T08:29:16.612608Z [info ] Delete InferenceService odh-mlserver-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:16.654389Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'sklearn', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:29:15Z', 'finalizers': ['odh.inferenceservice.finalizers', 'inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:15Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:29:15Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:29:15Z'}], 'name': 'odh-mlserver-image-isvc', 'namespace': 'runtime-verification', 'resourceVersion': '175446515', 'uid': '717572d3-74fd-40fb-bbcd-448e4526b0bc'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'sklearn'}, 'name': '', 'resources': {}, 'runtime': 'odh-mlserver-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}, 'status': {'address': {'url': 'http://odh-mlserver-image-isvc-predictor.runtime-verification.svc.cluster.local:8080'}, 'components': {'predictor': {}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:29:15Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:29:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:29:15Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:29:15Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0}, 'states': {'activeModelState': '', 'targetModelState': 'Pending'}, 'transitionStatus': 'InProgress'}, 'observedGeneration': 1, 'servingRuntimeName': 'odh-mlserver-image-runtime', 'url': 'http://odh-mlserver-image-isvc-predictor.runtime-verification.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:29:16.672320Z [info ] Wait until InferenceService odh-mlserver-image-isvc is deleted [ocp_resources InferenceService] 2026-06-11T08:29:16.672513Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:17.709246Z [info ] Elapsed time: 1.0170395374298096 [0:00:01.017040] [timeout_sampler] 2026-06-11T08:29:17.709458Z [info ] Delete ServingRuntime odh-mlserver-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:17.740891Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-11T08:29:15Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:15Z'}], 'name': 'odh-mlserver-image-runtime', 'namespace': 'runtime-verification', 'resourceVersion': '175446459', 'uid': 'b99d93d1-5228-4ffe-aed0-153e5caf357d'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:17.759545Z [info ] Wait until ServingRuntime odh-mlserver-image-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:29:17.759719Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:17.775015Z [info ] Elapsed time: 5.7220458984375e-05 [0:00:00.000057] [timeout_sampler] _ 4 of 17 completed, 4 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::TestServingRuntimeImagesPerTemplate::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime4]  --------------- test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime4] --------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:29:17.776541Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:17.776712Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:17.776866Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:17.777014Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:17.777189Z [info ] [cached since 30.29s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime4]', 1781166557) [sqlalchemy.engine.Engine] 2026-06-11T08:29:17.777347Z [info ] [cached since 30.29s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime4]', 1781166557) [sqlalchemy.engine.Engine] 2026-06-11T08:29:17.777802Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:17.777955Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:17.779259Z [info ] Executing function fixture: serving_runtime_pods_for_runtime [conftest] 2026-06-11T08:29:17.779440Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:29:17.813321Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:29:17.813546Z [info ] Create ServingRuntime rhaiis-vllm-cuda-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:17.813707Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v0.18.0', 'openshift.io/display-name': 'vLLM NVIDIA GPU ServingRuntime for KServe'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'rhaiis-vllm-cuda-image-runtime', 'namespace': 'runtime-verification'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'vllm', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8080'}, 'containers': [{'args': ['--port=8080', '--model=/mnt/models', '--served-model-name={{.Name}}'], 'command': ['python', '-m', 'vllm.entrypoints.openai.api_server'], 'env': [{'name': 'HF_HOME', 'value': '/tmp/hf_home'}], 'image': 'registry.redhat.io/rhaii/vllm-cuda-rhel9@sha256:1a799438f49926465794842fb0430b7ef8940c8f492ae19a0532248d1dcfb385', 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}]}], 'multiModel': False, 'supportedModelFormats': [{'autoSelect': True, 'name': 'vLLM'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:17.860281Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:29:17.860472Z [info ] Create InferenceService rhaiis-vllm-cuda-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:17.860668Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'rhaiis-vllm-cuda-image-isvc', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'runtime-verification'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'vLLM'}, 'version': None, 'runtime': 'rhaiis-vllm-cuda-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}} [ocp_resources InferenceService] 2026-06-11T08:29:17.905079Z [info ] Waiting for 300 seconds [0:05:00], retry every 5 seconds. (Function: timeout_sampler.wrapper Kwargs: {'client': , 'isvc': , 'runtime_name': 'rhaiis-vllm-cuda-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:17.905249Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'rhaiis-vllm-cuda-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:17.904488Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:18.964620Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:18.995210Z [info ] Elapsed time: 1.0601251125335693 [0:00:01.060125] [timeout_sampler] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-06-11T08:29:18.994567Z [info ] Validating rhaiis-vllm-cuda-image-isvc-predictor-794d7c5989-rbmqc in runtime-verification [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images PASSED2026-06-11T08:29:19.036557Z [info ] rhaiis_vllm_cuda_image : passed [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images  TEST: TestServingRuntimeImagesPerTemplate.test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime4] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-11T08:29:19.037974Z [info ] Elapsed time: 8.988380432128906e-05 [0:00:00.000090] [timeout_sampler] 2026-06-11T08:29:19.038117Z [info ] Delete InferenceService rhaiis-vllm-cuda-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:19.068564Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'vLLM', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:29:17Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:17Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:29:18Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:29:18Z'}], 'name': 'rhaiis-vllm-cuda-image-isvc', 'namespace': 'runtime-verification', 'resourceVersion': '175446766', 'uid': '836283b3-dbdf-4239-b7e3-fbbb01d9e868'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'vLLM'}, 'name': '', 'resources': {}, 'runtime': 'rhaiis-vllm-cuda-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}, 'status': {'address': {'url': 'http://rhaiis-vllm-cuda-image-isvc-predictor.runtime-verification.svc.cluster.local:8080'}, 'components': {'predictor': {}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:29:17Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:29:18Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:29:18Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:29:17Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0}, 'states': {'activeModelState': '', 'targetModelState': 'Pending'}, 'transitionStatus': 'InProgress'}, 'observedGeneration': 1, 'servingRuntimeName': 'rhaiis-vllm-cuda-image-runtime', 'url': 'http://rhaiis-vllm-cuda-image-isvc-predictor.runtime-verification.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:29:19.086798Z [info ] Wait until InferenceService rhaiis-vllm-cuda-image-isvc is deleted [ocp_resources InferenceService] 2026-06-11T08:29:19.086963Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:20.118048Z [info ] Elapsed time: 1.0154452323913574 [0:00:01.015445] [timeout_sampler] 2026-06-11T08:29:20.118203Z [info ] Delete ServingRuntime rhaiis-vllm-cuda-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:20.174615Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v0.18.0', 'openshift.io/display-name': 'vLLM NVIDIA GPU ServingRuntime for KServe'}, 'creationTimestamp': '2026-06-11T08:29:17Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/recommended-accelerators': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:17Z'}], 'name': 'rhaiis-vllm-cuda-image-runtime', 'namespace': 'runtime-verification', 'resourceVersion': '175446702', 'uid': 'e596243f-2762-4f11-b27e-6c6542ed5439'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'vllm', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8080'}, 'containers': [{'args': ['--port=8080', '--model=/mnt/models', '--served-model-name={{.Name}}'], 'command': ['python', '-m', 'vllm.entrypoints.openai.api_server'], 'env': [{'name': 'HF_HOME', 'value': '/tmp/hf_home'}], 'image': 'registry.redhat.io/rhaii/vllm-cuda-rhel9@sha256:1a799438f49926465794842fb0430b7ef8940c8f492ae19a0532248d1dcfb385', 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}]}], 'multiModel': False, 'supportedModelFormats': [{'autoSelect': True, 'name': 'vLLM'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:20.193372Z [info ] Wait until ServingRuntime rhaiis-vllm-cuda-image-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:29:20.193546Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:20.208051Z [info ] Elapsed time: 5.078315734863281e-05 [0:00:00.000051] [timeout_sampler] _ 5 of 17 completed, 5 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::TestServingRuntimeImagesPerTemplate::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime5]  --------------- test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime5] --------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:29:20.209592Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:20.209754Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:20.209908Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:20.210056Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:20.210225Z [info ] [cached since 32.72s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime5]', 1781166560) [sqlalchemy.engine.Engine] 2026-06-11T08:29:20.210419Z [info ] [cached since 32.72s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime5]', 1781166560) [sqlalchemy.engine.Engine] 2026-06-11T08:29:20.211108Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:20.212231Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:20.212426Z [info ] Executing function fixture: serving_runtime_pods_for_runtime [conftest] 2026-06-11T08:29:20.212557Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:29:20.246243Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:29:20.246717Z [info ] Create ServingRuntime rhaiis-vllm-rocm-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:20.246874Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["amd.com/gpu"]', 'opendatahub.io/runtime-version': 'v0.18.0', 'openshift.io/display-name': 'vLLM AMD GPU ServingRuntime for KServe'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'rhaiis-vllm-rocm-image-runtime', 'namespace': 'runtime-verification'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'vllm', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8080'}, 'containers': [{'args': ['--port=8080', '--model=/mnt/models', '--served-model-name={{.Name}}'], 'command': ['python', '-m', 'vllm.entrypoints.openai.api_server'], 'env': [{'name': 'HF_HOME', 'value': '/tmp/hf_home'}], 'image': 'registry.redhat.io/rhaii/vllm-rocm-rhel9@sha256:2d9b289cc7b24b19c80137e5b0a24515c6ba370f07e4b0fce48791d01ed1dc9a', 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}]}], 'multiModel': False, 'supportedModelFormats': [{'autoSelect': True, 'name': 'vLLM'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:20.323193Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:29:20.323324Z [info ] Create InferenceService rhaiis-vllm-rocm-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:20.323514Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'rhaiis-vllm-rocm-image-isvc', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'runtime-verification'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'vLLM'}, 'version': None, 'runtime': 'rhaiis-vllm-rocm-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}} [ocp_resources InferenceService] 2026-06-11T08:29:20.394862Z [info ] Waiting for 300 seconds [0:05:00], retry every 5 seconds. (Function: timeout_sampler.wrapper Kwargs: {'client': , 'isvc': , 'runtime_name': 'rhaiis-vllm-rocm-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:20.395020Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'rhaiis-vllm-rocm-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:20.394318Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:21.423462Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:21.454138Z [info ] Elapsed time: 1.0291194915771484 [0:00:01.029119] [timeout_sampler] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-06-11T08:29:21.453489Z [info ] Validating rhaiis-vllm-rocm-image-isvc-predictor-5755c5bc8f-tzvmm in runtime-verification [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images PASSED2026-06-11T08:29:21.495478Z [info ] rhaiis_vllm_rocm_image : passed [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images  TEST: TestServingRuntimeImagesPerTemplate.test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime5] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-11T08:29:21.496845Z [info ] Elapsed time: 5.91278076171875e-05 [0:00:00.000059] [timeout_sampler] 2026-06-11T08:29:21.496975Z [info ] Delete InferenceService rhaiis-vllm-rocm-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:21.530696Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'vLLM', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:29:20Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:20Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:29:20Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:29:20Z'}], 'name': 'rhaiis-vllm-rocm-image-isvc', 'namespace': 'runtime-verification', 'resourceVersion': '175446916', 'uid': '94a7f136-565a-4cbc-bdb4-357ccdeb0b8f'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'vLLM'}, 'name': '', 'resources': {}, 'runtime': 'rhaiis-vllm-rocm-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}, 'status': {'address': {'url': 'http://rhaiis-vllm-rocm-image-isvc-predictor.runtime-verification.svc.cluster.local:8080'}, 'components': {'predictor': {}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:29:20Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:29:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:29:20Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:29:20Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0}, 'states': {'activeModelState': '', 'targetModelState': 'Pending'}, 'transitionStatus': 'InProgress'}, 'observedGeneration': 1, 'servingRuntimeName': 'rhaiis-vllm-rocm-image-runtime', 'url': 'http://rhaiis-vllm-rocm-image-isvc-predictor.runtime-verification.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:29:21.549112Z [info ] Wait until InferenceService rhaiis-vllm-rocm-image-isvc is deleted [ocp_resources InferenceService] 2026-06-11T08:29:21.549250Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:22.578180Z [info ] Elapsed time: 1.0137848854064941 [0:00:01.013785] [timeout_sampler] 2026-06-11T08:29:22.578332Z [info ] Delete ServingRuntime rhaiis-vllm-rocm-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:22.613979Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["amd.com/gpu"]', 'opendatahub.io/runtime-version': 'v0.18.0', 'openshift.io/display-name': 'vLLM AMD GPU ServingRuntime for KServe'}, 'creationTimestamp': '2026-06-11T08:29:20Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/recommended-accelerators': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:20Z'}], 'name': 'rhaiis-vllm-rocm-image-runtime', 'namespace': 'runtime-verification', 'resourceVersion': '175446875', 'uid': '726449f1-f9d9-44d1-b103-dfd9274ccfe5'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'vllm', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8080'}, 'containers': [{'args': ['--port=8080', '--model=/mnt/models', '--served-model-name={{.Name}}'], 'command': ['python', '-m', 'vllm.entrypoints.openai.api_server'], 'env': [{'name': 'HF_HOME', 'value': '/tmp/hf_home'}], 'image': 'registry.redhat.io/rhaii/vllm-rocm-rhel9@sha256:2d9b289cc7b24b19c80137e5b0a24515c6ba370f07e4b0fce48791d01ed1dc9a', 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}]}], 'multiModel': False, 'supportedModelFormats': [{'autoSelect': True, 'name': 'vLLM'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:22.630688Z [info ] Wait until ServingRuntime rhaiis-vllm-rocm-image-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:29:22.630843Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:22.645109Z [info ] Elapsed time: 8.177757263183594e-05 [0:00:00.000082] [timeout_sampler] _ 6 of 17 completed, 6 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::TestServingRuntimeImagesPerTemplate::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime6]  --------------- test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime6] --------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:29:22.646573Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:22.646912Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:22.647062Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:22.648127Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:22.648739Z [info ] [cached since 35.16s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime6]', 1781166562) [sqlalchemy.engine.Engine] 2026-06-11T08:29:22.648887Z [info ] [cached since 35.16s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/image_validation/test_verify_serving_runtime_images.py::test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime6]', 1781166562) [sqlalchemy.engine.Engine] 2026-06-11T08:29:22.648996Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:22.649099Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:22.649198Z [info ] Executing function fixture: serving_runtime_pods_for_runtime [conftest] 2026-06-11T08:29:22.649298Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:29:22.698776Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:29:22.699042Z [info ] Create ServingRuntime rhaiis-vllm-spyre-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:22.699231Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["ibm.com/spyre_pf"]', 'opendatahub.io/runtime-version': 'v0.18.0', 'openshift.io/display-name': 'vLLM Spyre on x86 ServingRuntime for KServe'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'rhaiis-vllm-spyre-image-runtime', 'namespace': 'runtime-verification'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'vllm', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8080'}, 'containers': [{'args': ['--model=/mnt/models', '--port=8000', '--served-model-name={{.Name}}'], 'env': [{'name': 'HF_HOME', 'value': '/tmp/hf_home'}, {'name': 'FLEX_COMPUTE', 'value': 'SENTIENT'}, {'name': 'FLEX_DEVICE', 'value': 'PF'}, {'name': 'TOKENIZERS_PARALLELISM', 'value': 'false'}, {'name': 'DTLOG_LEVEL', 'value': 'error'}, {'name': 'TORCH_SENDNN_LOG', 'value': 'CRITICAL'}, {'name': 'VLLM_SPYRE_WARMUP_BATCH_SIZES', 'value': '4'}, {'name': 'VLLM_SPYRE_WARMUP_PROMPT_LENS', 'value': '1024'}, {'name': 'VLLM_SPYRE_WARMUP_NEW_TOKENS', 'value': '256'}, {'name': 'VLLM_SPYRE_REQUIRE_PRECOMPILED_DECODERS', 'value': '0'}], 'image': 'registry.redhat.io/rhaii/vllm-spyre-rhel9@sha256:26a609247b8271a6f408e3c23adf95f15d2fe4e1d9befcfef5b6a06cfa6d717b', 'name': 'kserve-container', 'ports': [{'containerPort': 8000, 'protocol': 'TCP'}], 'volumeMounts': [{'mountPath': '/opt/ibm/spyre/models', 'name': 'spyre-cache'}]}], 'multiModel': False, 'schedulerName': 'spyre-scheduler', 'supportedModelFormats': [{'autoSelect': True, 'name': 'vLLM'}], 'volumes': [{'emptyDir': {}, 'name': 'spyre-cache'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:22.776969Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:29:22.777383Z [info ] Create InferenceService rhaiis-vllm-spyre-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:22.777532Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'rhaiis-vllm-spyre-image-isvc', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'runtime-verification'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'vLLM'}, 'version': None, 'runtime': 'rhaiis-vllm-spyre-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}} [ocp_resources InferenceService] 2026-06-11T08:29:22.891202Z [info ] Waiting for 300 seconds [0:05:00], retry every 5 seconds. (Function: timeout_sampler.wrapper Kwargs: {'client': , 'isvc': , 'runtime_name': 'rhaiis-vllm-spyre-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:22.891439Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'rhaiis-vllm-spyre-image-runtime'}) [timeout_sampler] 2026-06-11T08:29:22.890322Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:23.918143Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:29:23.968386Z [info ] Elapsed time: 1.027876615524292 [0:00:01.027877] [timeout_sampler] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-06-11T08:29:23.967456Z [info ] Validating rhaiis-vllm-spyre-image-isvc-predictor-57cbdb9669-bcqll in runtime-verification [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images PASSED2026-06-11T08:29:24.009095Z [info ] rhaiis_vllm_spyre_image : passed [_pytest.compat] name=tests.model_serving.model_runtime.image_validation.test_verify_serving_runtime_images  TEST: TestServingRuntimeImagesPerTemplate.test_verify_serving_runtime_pod_images_from_template[serving_runtime_pods_for_runtime6] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-11T08:29:24.011575Z [info ] Elapsed time: 0.0001735687255859375 [0:00:00.000174] [timeout_sampler] 2026-06-11T08:29:24.011704Z [info ] Delete InferenceService rhaiis-vllm-spyre-image-isvc [ocp_resources InferenceService] 2026-06-11T08:29:24.076683Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'vLLM', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:29:22Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:22Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:29:23Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:29:23Z'}], 'name': 'rhaiis-vllm-spyre-image-isvc', 'namespace': 'runtime-verification', 'resourceVersion': '175447050', 'uid': '2bfea4fb-0bfb-4051-b399-6cf41e9da0a3'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'vLLM'}, 'name': '', 'resources': {}, 'runtime': 'rhaiis-vllm-spyre-image-runtime', 'storageUri': 's3://dummy-bucket/dummy/'}}}, 'status': {'address': {'url': 'http://rhaiis-vllm-spyre-image-isvc-predictor.runtime-verification.svc.cluster.local:8080'}, 'components': {'predictor': {}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:29:22Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:29:23Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:29:23Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:29:22Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0}, 'states': {'activeModelState': '', 'targetModelState': 'Pending'}, 'transitionStatus': 'InProgress'}, 'observedGeneration': 1, 'servingRuntimeName': 'rhaiis-vllm-spyre-image-runtime', 'url': 'http://rhaiis-vllm-spyre-image-isvc-predictor.runtime-verification.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:29:24.096791Z [info ] Wait until InferenceService rhaiis-vllm-spyre-image-isvc is deleted [ocp_resources InferenceService] 2026-06-11T08:29:24.097074Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:25.160975Z [info ] Elapsed time: 1.0481970310211182 [0:00:01.048197] [timeout_sampler] 2026-06-11T08:29:25.161412Z [info ] Delete ServingRuntime rhaiis-vllm-spyre-image-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:25.192307Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["ibm.com/spyre_pf"]', 'opendatahub.io/runtime-version': 'v0.18.0', 'openshift.io/display-name': 'vLLM Spyre on x86 ServingRuntime for KServe'}, 'creationTimestamp': '2026-06-11T08:29:22Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/recommended-accelerators': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:schedulerName': {}, 'f:supportedModelFormats': {}, 'f:volumes': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:22Z'}], 'name': 'rhaiis-vllm-spyre-image-runtime', 'namespace': 'runtime-verification', 'resourceVersion': '175447005', 'uid': '5b3ca6c0-e45b-4c58-96f2-23bc6f05c1e3'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'vllm', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8080'}, 'containers': [{'args': ['--model=/mnt/models', '--port=8000', '--served-model-name={{.Name}}'], 'env': [{'name': 'HF_HOME', 'value': '/tmp/hf_home'}, {'name': 'FLEX_COMPUTE', 'value': 'SENTIENT'}, {'name': 'FLEX_DEVICE', 'value': 'PF'}, {'name': 'TOKENIZERS_PARALLELISM', 'value': 'false'}, {'name': 'DTLOG_LEVEL', 'value': 'error'}, {'name': 'TORCH_SENDNN_LOG', 'value': 'CRITICAL'}, {'name': 'VLLM_SPYRE_WARMUP_BATCH_SIZES', 'value': '4'}, {'name': 'VLLM_SPYRE_WARMUP_PROMPT_LENS', 'value': '1024'}, {'name': 'VLLM_SPYRE_WARMUP_NEW_TOKENS', 'value': '256'}, {'name': 'VLLM_SPYRE_REQUIRE_PRECOMPILED_DECODERS', 'value': '0'}], 'image': 'registry.redhat.io/rhaii/vllm-spyre-rhel9@sha256:26a609247b8271a6f408e3c23adf95f15d2fe4e1d9befcfef5b6a06cfa6d717b', 'name': 'kserve-container', 'ports': [{'containerPort': 8000, 'protocol': 'TCP'}], 'volumeMounts': [{'mountPath': '/opt/ibm/spyre/models', 'name': 'spyre-cache'}]}], 'multiModel': False, 'schedulerName': 'spyre-scheduler', 'supportedModelFormats': [{'autoSelect': True, 'name': 'vLLM'}], 'volumes': [{'emptyDir': {}, 'name': 'spyre-cache'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:25.220120Z [info ] Wait until ServingRuntime rhaiis-vllm-spyre-image-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:29:25.220417Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:25.235167Z [info ] Elapsed time: 7.081031799316406e-05 [0:00:00.000071] [timeout_sampler] 2026-06-11T08:29:25.283181Z [info ] Delete Namespace runtime-verification [ocp_resources Namespace] 2026-06-11T08:29:25.327332Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'runtime-verification', 'uid': '39a0f074-3b2f-4c0d-b948-20ff1f68b81b', 'resourceVersion': '175445828', 'creationTimestamp': '2026-06-11T08:29:07Z', 'labels': {'kubernetes.io/metadata.name': 'runtime-verification', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c33,c12', 'openshift.io/sa.scc.supplemental-groups': '1001080000/10000', 'openshift.io/sa.scc.uid-range': '1001080000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-11T08:29:07Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:29:07Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:29:07Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-11T08:29:25.347081Z [info ] Wait until Namespace runtime-verification is deleted [ocp_resources Namespace] 2026-06-11T08:29:25.347442Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] _ 7 of 17 completed, 7 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment] 2026-06-11T08:29:34.563617Z [info ] Elapsed time: 9.16510796546936 [0:00:09.165108] [timeout_sampler]  -------------------------- test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment] -------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:29:34.568186Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:34.568422Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:29:34.568574Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:34.568700Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:29:34.568872Z [info ] [cached since 47.08s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment]', 1781166574) [sqlalchemy.engine.Engine] 2026-06-11T08:29:34.569029Z [info ] [cached since 47.08s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment]', 1781166574) [sqlalchemy.engine.Engine] 2026-06-11T08:29:34.569875Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:34.570006Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:29:34.573215Z [info ] Executing session fixture: pytestconfig [conftest] 2026-06-11T08:29:34.573436Z [info ] Executing session fixture: teardown_resources [conftest] 2026-06-11T08:29:34.573592Z [info ] Executing session fixture: mlserver_runtime_image [conftest] 2026-06-11T08:29:34.573718Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-11T08:29:34.573838Z [info ] Create Namespace sklearn-modelcar [ocp_resources Namespace] 2026-06-11T08:29:34.573960Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'sklearn-modelcar'}, 'spec': {}} [ocp_resources Namespace] 2026-06-11T08:29:34.648760Z [info ] Wait for Namespace sklearn-modelcar status to be Active [ocp_resources Namespace] 2026-06-11T08:29:34.648902Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:34.698529Z [info ] Status of Namespace sklearn-modelcar is Active [ocp_resources Namespace] 2026-06-11T08:29:34.698742Z [info ] Elapsed time: 3.910064697265625e-05 [0:00:00.000039] [timeout_sampler] 2026-06-11T08:29:34.698859Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-11T08:29:34.698972Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:29:34.772593Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:29:34.772800Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-11T08:29:34.773570Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'sklearn-modelcar'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:29:34.861867Z [info ] Executing class fixture: mlserver_model_car_inference_service [conftest] 2026-06-11T08:29:34.862258Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:29:34.862590Z [info ] Create InferenceService sklearn-modelcar [ocp_resources InferenceService] 2026-06-11T08:29:34.862761Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'sklearn-modelcar', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'sklearn-modelcar'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'sklearn'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55'}}}} [ocp_resources InferenceService] 2026-06-11T08:29:34.930954Z [info ] Wait for InferenceService/sklearn-modelcar's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-11T08:29:34.931334Z [info ] Wait until InferenceService sklearn-modelcar is created [ocp_resources InferenceService] 2026-06-11T08:29:34.931652Z [info ] Waiting for 899.9999623298645 seconds [0:14:59.999962], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-11T08:29:34.946083Z [info ] Elapsed time: 4.673004150390625e-05 [0:00:00.000047] [timeout_sampler] 2026-06-11T08:29:34.946421Z [info ] Waiting for 899.9848034381866 seconds [0:14:59.984803], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-11T08:30:46.037698Z [info ] Elapsed time: 71.06384086608887 [0:01:11.063841] [timeout_sampler] 2026-06-11T08:30:46.068504Z [info ] Waiting for 828.8630573749542 seconds [0:13:48.863057], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-11T08:30:46.125754Z [info ] Elapsed time: 0.00022602081298828125 [0:00:00.000226] [timeout_sampler] 2026-06-11T08:30:46.141176Z [info ] Executing function fixture: snapshot [conftest] 2026-06-11T08:30:46.142200Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-11T08:30:46Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED _ 8 of 17 completed, 8 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[xgboost-modelcar-RawDeployment]  TEST: TestMLServerModelCar.test_mlserver_model_car_inference[sklearn-modelcar-RawDeployment] STATUS: PASSED ------------------------------------------------------- TEARDOWN -------------------------------------------------------  -------------------------- test_mlserver_model_car_inference[xgboost-modelcar-RawDeployment] -------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:30:46.487818Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:30:46.489029Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:30:46.489457Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:30:46.489761Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:30:46.490001Z [info ] [cached since 119s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[xgboost-modelcar-RawDeployment]', 1781166646) [sqlalchemy.engine.Engine] 2026-06-11T08:30:46.490143Z [info ] [cached since 119s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[xgboost-modelcar-RawDeployment]', 1781166646) [sqlalchemy.engine.Engine] 2026-06-11T08:30:46.490261Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:30:46.490394Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:30:46.490506Z [info ] Delete InferenceService sklearn-modelcar [ocp_resources InferenceService] 2026-06-11T08:30:46.523135Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'sklearn', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:29:34Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:34Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:29:35Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:30:45Z'}], 'name': 'sklearn-modelcar', 'namespace': 'sklearn-modelcar', 'resourceVersion': '175449712', 'uid': 'c6a37cc3-3455-43e1-9a2b-d36e3b24ee6d'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'sklearn'}, 'name': '', 'resources': {}, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-sklearn@sha256:ec9bc6b520909c52bd1d4accc2b2d28adb04981bd4c3ce94f17f23dd573e1f55'}}}, 'status': {'address': {'url': 'http://sklearn-modelcar-predictor.sklearn-modelcar.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://sklearn-modelcar-predictor-sklearn-modelcar.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:29:35Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:30:45Z', 'message': 'ReplicaSet "sklearn-modelcar-predictor-77dbd84c86" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:30:45Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:29:35Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://sklearn-modelcar-predictor.sklearn-modelcar.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:30:46.542833Z [info ] Wait until InferenceService sklearn-modelcar is deleted [ocp_resources InferenceService] 2026-06-11T08:30:46.543015Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:30:47.576103Z [info ] Elapsed time: 1.014068841934204 [0:00:01.014069] [timeout_sampler] 2026-06-11T08:30:47.576269Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-11T08:30:47.607192Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-11T08:29:34Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:29:34Z'}], 'name': 'mlserver-runtime', 'namespace': 'sklearn-modelcar', 'resourceVersion': '175447553', 'uid': '6c30fcff-9a37-499c-a946-a92a063f742e'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:30:47.624269Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:30:47.624440Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:30:47.638433Z [info ] Elapsed time: 6.318092346191406e-05 [0:00:00.000063] [timeout_sampler] 2026-06-11T08:30:47.691179Z [info ] Delete Namespace sklearn-modelcar [ocp_resources Namespace] 2026-06-11T08:30:47.721769Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'sklearn-modelcar', 'uid': 'd874e27d-db1e-4976-a277-5a9528d6817e', 'resourceVersion': '175447532', 'creationTimestamp': '2026-06-11T08:29:34Z', 'labels': {'kubernetes.io/metadata.name': 'sklearn-modelcar', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c33,c17', 'openshift.io/sa.scc.supplemental-groups': '1001090000/10000', 'openshift.io/sa.scc.uid-range': '1001090000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-11T08:29:34Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:29:34Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:29:34Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-11T08:30:47.744879Z [info ] Wait until Namespace sklearn-modelcar is deleted [ocp_resources Namespace] 2026-06-11T08:30:47.745060Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:30:56.195325Z [info ] Elapsed time: 8.433596134185791 [0:00:08.433596] [timeout_sampler] 2026-06-11T08:30:56.195692Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-11T08:30:56.195826Z [info ] Create Namespace xgboost-modelcar [ocp_resources Namespace] 2026-06-11T08:30:56.195947Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'xgboost-modelcar'}, 'spec': {}} [ocp_resources Namespace] 2026-06-11T08:30:56.242916Z [info ] Wait for Namespace xgboost-modelcar status to be Active [ocp_resources Namespace] 2026-06-11T08:30:56.243066Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-11T08:30:56.264948Z [info ] Status of Namespace xgboost-modelcar is Active [ocp_resources Namespace] 2026-06-11T08:30:56.265231Z [info ] Elapsed time: 3.7670135498046875e-05 [0:00:00.000038] [timeout_sampler] 2026-06-11T08:30:56.265367Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-11T08:30:56.265491Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:30:56.317053Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:30:56.317316Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-11T08:30:56.317482Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'xgboost-modelcar'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:30:56.360800Z [info ] Executing class fixture: mlserver_model_car_inference_service [conftest] 2026-06-11T08:30:56.361181Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:30:56.361318Z [info ] Create InferenceService xgboost-modelcar [ocp_resources InferenceService] 2026-06-11T08:30:56.361470Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'xgboost-modelcar', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'xgboost-modelcar'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'xgboost'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-xgboost@sha256:5b6982bdc939b53a7a1210f56aa52bf7de0f0cbc693668db3fd1f496571bff29'}}}} [ocp_resources InferenceService] 2026-06-11T08:30:56.438150Z [info ] Wait for InferenceService/xgboost-modelcar's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-11T08:30:56.438323Z [info ] Wait until InferenceService xgboost-modelcar is created [ocp_resources InferenceService] 2026-06-11T08:30:56.438471Z [info ] Waiting for 899.9999933242798 seconds [0:14:59.999993], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-11T08:30:56.452917Z [info ] Elapsed time: 3.4809112548828125e-05 [0:00:00.000035] [timeout_sampler] 2026-06-11T08:30:56.453059Z [info ] Waiting for 899.9849944114685 seconds [0:14:59.984994], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-11T08:31:14.149856Z [info ] Elapsed time: 17.682345867156982 [0:00:17.682346] [timeout_sampler] 2026-06-11T08:31:14.163229Z [info ] Waiting for 882.2747566699982 seconds [0:14:42.274757], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-11T08:31:14.176575Z [info ] Elapsed time: 7.152557373046875e-05 [0:00:00.000072] [timeout_sampler] 2026-06-11T08:31:14.191666Z [info ] Executing function fixture: snapshot [conftest] 2026-06-11T08:31:14.191854Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-11T08:31:14Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED _ 9 of 17 completed, 9 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment]  TEST: TestMLServerModelCar.test_mlserver_model_car_inference[xgboost-modelcar-RawDeployment] STATUS: PASSED ------------------------------------------------------- TEARDOWN -------------------------------------------------------  -------------------------- test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment] -------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:31:14.436432Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:31:14.436574Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:31:14.436706Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:31:14.436825Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:31:14.436974Z [info ] [cached since 146.9s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment]', 1781166674) [sqlalchemy.engine.Engine] 2026-06-11T08:31:14.437114Z [info ] [cached since 146.9s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment]', 1781166674) [sqlalchemy.engine.Engine] 2026-06-11T08:31:14.438497Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:31:14.438730Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:31:14.438861Z [info ] Delete InferenceService xgboost-modelcar [ocp_resources InferenceService] 2026-06-11T08:31:14.471284Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'xgboost', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:30:56Z', 'finalizers': ['odh.inferenceservice.finalizers', 'inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:30:56Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:30:56Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:31:13Z'}], 'name': 'xgboost-modelcar', 'namespace': 'xgboost-modelcar', 'resourceVersion': '175450622', 'uid': '0f640c5f-8c5d-49ea-a032-a01e1dac6fe3'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'xgboost'}, 'name': '', 'resources': {}, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-xgboost@sha256:5b6982bdc939b53a7a1210f56aa52bf7de0f0cbc693668db3fd1f496571bff29'}}}, 'status': {'address': {'url': 'http://xgboost-modelcar-predictor.xgboost-modelcar.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://xgboost-modelcar-predictor-xgboost-modelcar.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:30:56Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:31:13Z', 'message': 'ReplicaSet "xgboost-modelcar-predictor-6fdfc778d8" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:31:13Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:30:56Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://xgboost-modelcar-predictor.xgboost-modelcar.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:31:14.515422Z [info ] Wait until InferenceService xgboost-modelcar is deleted [ocp_resources InferenceService] 2026-06-11T08:31:14.515608Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:31:15.545422Z [info ] Elapsed time: 1.0143179893493652 [0:00:01.014318] [timeout_sampler] 2026-06-11T08:31:15.545771Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-11T08:31:15.619447Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-11T08:30:56Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:30:56Z'}], 'name': 'mlserver-runtime', 'namespace': 'xgboost-modelcar', 'resourceVersion': '175450128', 'uid': '98262c55-89c4-4e48-aa22-8ec269b191ef'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:31:15.637418Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:31:15.637742Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:31:15.651716Z [info ] Elapsed time: 8.153915405273438e-05 [0:00:00.000082] [timeout_sampler] 2026-06-11T08:31:15.705857Z [info ] Delete Namespace xgboost-modelcar [ocp_resources Namespace] 2026-06-11T08:31:15.736832Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'xgboost-modelcar', 'uid': '57ccbe9c-81c2-4c96-bf11-7ea5b653cdeb', 'resourceVersion': '175450116', 'creationTimestamp': '2026-06-11T08:30:56Z', 'labels': {'kubernetes.io/metadata.name': 'xgboost-modelcar', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c33,c22', 'openshift.io/sa.scc.supplemental-groups': '1001100000/10000', 'openshift.io/sa.scc.uid-range': '1001100000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-11T08:30:56Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:30:56Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:30:56Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-11T08:31:15.757647Z [info ] Wait until Namespace xgboost-modelcar is deleted [ocp_resources Namespace] 2026-06-11T08:31:15.757888Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:31:23.918634Z [info ] Elapsed time: 8.144448518753052 [0:00:08.144449] [timeout_sampler] 2026-06-11T08:31:23.919080Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-11T08:31:23.919218Z [info ] Create Namespace lightgbm-modelcar [ocp_resources Namespace] 2026-06-11T08:31:23.919340Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'lightgbm-modelcar'}, 'spec': {}} [ocp_resources Namespace] 2026-06-11T08:31:23.975258Z [info ] Wait for Namespace lightgbm-modelcar status to be Active [ocp_resources Namespace] 2026-06-11T08:31:23.975437Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-11T08:31:23.992103Z [info ] Status of Namespace lightgbm-modelcar is Active [ocp_resources Namespace] 2026-06-11T08:31:23.992263Z [info ] Elapsed time: 0.00015234947204589844 [0:00:00.000152] [timeout_sampler] 2026-06-11T08:31:23.992413Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-11T08:31:23.992535Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:31:24.049600Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:31:24.049745Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-11T08:31:24.049907Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'lightgbm-modelcar'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:31:24.143975Z [info ] Executing class fixture: mlserver_model_car_inference_service [conftest] 2026-06-11T08:31:24.144186Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:31:24.144318Z [info ] Create InferenceService lightgbm-modelcar [ocp_resources InferenceService] 2026-06-11T08:31:24.144476Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'lightgbm-modelcar', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'lightgbm-modelcar'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'lightgbm'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-lightgbm@sha256:77eb15a2eccefa3756faaf2ee4bc1e63990b746427d323957c461f33a4f1a6a3'}}}} [ocp_resources InferenceService] 2026-06-11T08:31:24.208115Z [info ] Wait for InferenceService/lightgbm-modelcar's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-11T08:31:24.208338Z [info ] Wait until InferenceService lightgbm-modelcar is created [ocp_resources InferenceService] 2026-06-11T08:31:24.208483Z [info ] Waiting for 899.9999964237213 seconds [0:14:59.999996], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-11T08:31:24.243243Z [info ] Elapsed time: 9.34600830078125e-05 [0:00:00.000093] [timeout_sampler] 2026-06-11T08:31:24.243435Z [info ] Waiting for 899.9646451473236 seconds [0:14:59.964645], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-11T08:31:40.567435Z [info ] Elapsed time: 16.309139013290405 [0:00:16.309139] [timeout_sampler] 2026-06-11T08:31:40.581853Z [info ] Waiting for 883.6256291866302 seconds [0:14:43.625629], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-11T08:31:40.596458Z [info ] Elapsed time: 0.0001354217529296875 [0:00:00.000135] [timeout_sampler] 2026-06-11T08:31:40.610688Z [info ] Executing function fixture: snapshot [conftest] 2026-06-11T08:31:40.610828Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-11T08:31:40Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED _ 10 of 17 completed, 10 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment_text_type]  TEST: TestMLServerModelCar.test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment] STATUS: PASSED ------------------------------------------------------- TEARDOWN -------------------------------------------------------  --------------------- test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment_text_type] --------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:31:40.857854Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:31:40.857979Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:31:40.858102Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:31:40.858221Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:31:40.858625Z [info ] [cached since 173.4s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment_text_type]', 1781166700) [sqlalchemy.engine.Engine] 2026-06-11T08:31:40.858795Z [info ] [cached since 173.4s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment_text_type]', 1781166700) [sqlalchemy.engine.Engine] 2026-06-11T08:31:40.858923Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:31:40.859041Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:31:40.860018Z [info ] Delete InferenceService lightgbm-modelcar [ocp_resources InferenceService] 2026-06-11T08:31:40.892259Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'lightgbm', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:31:24Z', 'finalizers': ['odh.inferenceservice.finalizers', 'inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:31:24Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:31:24Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:31:40Z'}], 'name': 'lightgbm-modelcar', 'namespace': 'lightgbm-modelcar', 'resourceVersion': '175451497', 'uid': '3fd3fbe6-effb-4586-8588-e007afba111c'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'lightgbm'}, 'name': '', 'resources': {}, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-lightgbm@sha256:77eb15a2eccefa3756faaf2ee4bc1e63990b746427d323957c461f33a4f1a6a3'}}}, 'status': {'address': {'url': 'http://lightgbm-modelcar-predictor.lightgbm-modelcar.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://lightgbm-modelcar-predictor-lightgbm-modelcar.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:31:24Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:31:40Z', 'message': 'ReplicaSet "lightgbm-modelcar-predictor-84f4b8f9db" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:31:40Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:31:24Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://lightgbm-modelcar-predictor.lightgbm-modelcar.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:31:40.911527Z [info ] Wait until InferenceService lightgbm-modelcar is deleted [ocp_resources InferenceService] 2026-06-11T08:31:40.911678Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:31:41.942251Z [info ] Elapsed time: 1.0146846771240234 [0:00:01.014685] [timeout_sampler] 2026-06-11T08:31:41.942572Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-11T08:31:41.975157Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-11T08:31:24Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:31:24Z'}], 'name': 'mlserver-runtime', 'namespace': 'lightgbm-modelcar', 'resourceVersion': '175451013', 'uid': '85eff601-7307-444a-8f44-07f4dbc0dd33'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:31:41.993038Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:31:41.993439Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:31:42.007864Z [info ] Elapsed time: 7.963180541992188e-05 [0:00:00.000080] [timeout_sampler] 2026-06-11T08:31:42.081037Z [info ] Delete Namespace lightgbm-modelcar [ocp_resources Namespace] 2026-06-11T08:31:42.134575Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'lightgbm-modelcar', 'uid': 'aec82fc4-ecc0-4e0a-8954-fb728ff0db49', 'resourceVersion': '175450994', 'creationTimestamp': '2026-06-11T08:31:23Z', 'labels': {'kubernetes.io/metadata.name': 'lightgbm-modelcar', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c33,c27', 'openshift.io/sa.scc.supplemental-groups': '1001110000/10000', 'openshift.io/sa.scc.uid-range': '1001110000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-11T08:31:23Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:31:23Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:31:23Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-11T08:31:42.156938Z [info ] Wait until Namespace lightgbm-modelcar is deleted [ocp_resources Namespace] 2026-06-11T08:31:42.157093Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:31:50.649337Z [info ] Elapsed time: 8.418396949768066 [0:00:08.418397] [timeout_sampler] 2026-06-11T08:31:50.649683Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-11T08:31:50.649866Z [info ] Create Namespace lightgbm-model-car-text-type [ocp_resources Namespace] 2026-06-11T08:31:50.650038Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'lightgbm-model-car-text-type'}, 'spec': {}} [ocp_resources Namespace] 2026-06-11T08:31:50.731855Z [info ] Wait for Namespace lightgbm-model-car-text-type status to be Active [ocp_resources Namespace] 2026-06-11T08:31:50.732011Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-11T08:31:50.750889Z [info ] Status of Namespace lightgbm-model-car-text-type is Active [ocp_resources Namespace] 2026-06-11T08:31:50.751045Z [info ] Elapsed time: 9.083747863769531e-05 [0:00:00.000091] [timeout_sampler] 2026-06-11T08:31:50.751156Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-11T08:31:50.751272Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:31:50.839286Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:31:50.839755Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-11T08:31:50.839997Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'lightgbm-model-car-text-type'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:31:50.892876Z [info ] Executing class fixture: mlserver_model_car_inference_service [conftest] 2026-06-11T08:31:50.893064Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:31:50.893195Z [info ] Create InferenceService lightgbm-modelcar [ocp_resources InferenceService] 2026-06-11T08:31:50.893323Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'lightgbm-modelcar', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'lightgbm-model-car-text-type'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'lightgbm'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-lightgbm@sha256:77eb15a2eccefa3756faaf2ee4bc1e63990b746427d323957c461f33a4f1a6a3', 'env': [{'name': 'MLSERVER_MODEL_URI', 'value': '/mnt/models/model.txt'}]}}}} [ocp_resources InferenceService] 2026-06-11T08:31:50.937122Z [info ] Wait for InferenceService/lightgbm-modelcar's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-11T08:31:50.937284Z [info ] Wait until InferenceService lightgbm-modelcar is created [ocp_resources InferenceService] 2026-06-11T08:31:50.937441Z [info ] Waiting for 899.9999964237213 seconds [0:14:59.999996], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-11T08:31:51.001221Z [info ] Elapsed time: 0.00010085105895996094 [0:00:00.000101] [timeout_sampler] 2026-06-11T08:31:51.001430Z [info ] Waiting for 899.9353129863739 seconds [0:14:59.935313], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-11T08:32:03.403811Z [info ] Elapsed time: 12.384735822677612 [0:00:12.384736] [timeout_sampler] 2026-06-11T08:32:03.417958Z [info ] Waiting for 887.5184695720673 seconds [0:14:47.518470], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-11T08:32:03.432531Z [info ] Elapsed time: 7.796287536621094e-05 [0:00:00.000078] [timeout_sampler] 2026-06-11T08:32:03.447842Z [info ] Executing function fixture: snapshot [conftest] 2026-06-11T08:32:03.448008Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-11T08:32:04Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED _ 11 of 17 completed, 11 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::TestMLServerModelCar::test_mlserver_model_car_inference[onnx-modelcar-RawDeployment]  TEST: TestMLServerModelCar.test_mlserver_model_car_inference[lightgbm-modelcar-RawDeployment_text_type] STATUS: PASSED ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ---------------------------- test_mlserver_model_car_inference[onnx-modelcar-RawDeployment] ---------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:32:05.103706Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:32:05.103839Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:32:05.103962Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:32:05.104119Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:32:05.104554Z [info ] [cached since 197.6s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[onnx-modelcar-RawDeployment]', 1781166725) [sqlalchemy.engine.Engine] 2026-06-11T08:32:05.104701Z [info ] [cached since 197.6s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/model_car/test_mlserver_model_car.py::test_mlserver_model_car_inference[onnx-modelcar-RawDeployment]', 1781166725) [sqlalchemy.engine.Engine] 2026-06-11T08:32:05.104825Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:32:05.104944Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:32:05.106013Z [info ] Delete InferenceService lightgbm-modelcar [ocp_resources InferenceService] 2026-06-11T08:32:05.141141Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'lightgbm', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:31:50Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:env': {'.': {}, 'k:{"name":"MLSERVER_MODEL_URI"}': {'.': {}, 'f:name': {}, 'f:value': {}}}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:31:50Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:31:51Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:32:02Z'}], 'name': 'lightgbm-modelcar', 'namespace': 'lightgbm-model-car-text-type', 'resourceVersion': '175452236', 'uid': 'bdcbd83c-8b30-4558-9a1a-0d99334e29d5'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'env': [{'name': 'MLSERVER_MODEL_URI', 'value': '/mnt/models/model.txt'}], 'modelFormat': {'name': 'lightgbm'}, 'name': '', 'resources': {}, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-lightgbm@sha256:77eb15a2eccefa3756faaf2ee4bc1e63990b746427d323957c461f33a4f1a6a3'}}}, 'status': {'address': {'url': 'http://lightgbm-modelcar-predictor.lightgbm-model-car-text-type.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://lightgbm-modelcar-predictor-lightgbm-model-car-text-type.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:31:51Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:32:02Z', 'message': 'ReplicaSet "lightgbm-modelcar-predictor-7fccb6999d" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:32:02Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:31:51Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://lightgbm-modelcar-predictor.lightgbm-model-car-text-type.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:32:05.173075Z [info ] Wait until InferenceService lightgbm-modelcar is deleted [ocp_resources InferenceService] 2026-06-11T08:32:05.173414Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:06.204434Z [info ] Elapsed time: 1.0156712532043457 [0:00:01.015671] [timeout_sampler] 2026-06-11T08:32:06.204606Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-11T08:32:06.240415Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-11T08:31:50Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:31:50Z'}], 'name': 'mlserver-runtime', 'namespace': 'lightgbm-model-car-text-type', 'resourceVersion': '175451887', 'uid': 'bd9ed35c-cfe8-4355-bf28-84fe714f204c'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:32:06.272030Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:32:06.272291Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:06.292676Z [info ] Elapsed time: 0.00010251998901367188 [0:00:00.000103] [timeout_sampler] 2026-06-11T08:32:06.658694Z [info ] Delete Namespace lightgbm-model-car-text-type [ocp_resources Namespace] 2026-06-11T08:32:06.693393Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'lightgbm-model-car-text-type', 'uid': '968aa287-3c63-4263-b318-d152ae230cf5', 'resourceVersion': '175451866', 'creationTimestamp': '2026-06-11T08:31:50Z', 'labels': {'kubernetes.io/metadata.name': 'lightgbm-model-car-text-type', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c33,c32', 'openshift.io/sa.scc.supplemental-groups': '1001120000/10000', 'openshift.io/sa.scc.uid-range': '1001120000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-11T08:31:50Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:31:50Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:31:50Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-11T08:32:06.710447Z [info ] Wait until Namespace lightgbm-model-car-text-type is deleted [ocp_resources Namespace] 2026-06-11T08:32:06.710683Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:14.854929Z [info ] Elapsed time: 8.12816071510315 [0:00:08.128161] [timeout_sampler] 2026-06-11T08:32:14.855167Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-11T08:32:14.855331Z [info ] Create Namespace onnx-modelcar [ocp_resources Namespace] 2026-06-11T08:32:14.855547Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'onnx-modelcar'}, 'spec': {}} [ocp_resources Namespace] 2026-06-11T08:32:14.895416Z [info ] Wait for Namespace onnx-modelcar status to be Active [ocp_resources Namespace] 2026-06-11T08:32:14.895601Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:14.911098Z [info ] Status of Namespace onnx-modelcar is Active [ocp_resources Namespace] 2026-06-11T08:32:14.911297Z [info ] Elapsed time: 4.5299530029296875e-05 [0:00:00.000045] [timeout_sampler] 2026-06-11T08:32:14.911476Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-11T08:32:14.911615Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:32:14.959851Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:32:14.960014Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-11T08:32:14.960159Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'onnx-modelcar'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:32:14.996679Z [info ] Executing class fixture: mlserver_model_car_inference_service [conftest] 2026-06-11T08:32:14.996955Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:32:14.997473Z [info ] Create InferenceService onnx-modelcar [ocp_resources InferenceService] 2026-06-11T08:32:14.997967Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'onnx-modelcar', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'onnx-modelcar'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'onnx'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-onnx@sha256:d0ad00fb6f2caa8f02a0250fc44a576771d0846b2ac8d164ec203b10ec5d604b'}}}} [ocp_resources InferenceService] 2026-06-11T08:32:15.057410Z [info ] Wait for InferenceService/onnx-modelcar's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-11T08:32:15.057643Z [info ] Wait until InferenceService onnx-modelcar is created [ocp_resources InferenceService] 2026-06-11T08:32:15.057774Z [info ] Waiting for 899.9999973773956 seconds [0:14:59.999997], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:15.071922Z [info ] Elapsed time: 5.650520324707031e-05 [0:00:00.000057] [timeout_sampler] 2026-06-11T08:32:15.072076Z [info ] Waiting for 899.9854252338409 seconds [0:14:59.985425], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-11T08:32:28.310774Z [info ] Elapsed time: 13.224040269851685 [0:00:13.224040] [timeout_sampler] 2026-06-11T08:32:28.327216Z [info ] Waiting for 886.7300028800964 seconds [0:14:46.730003], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-11T08:32:28.341239Z [info ] Elapsed time: 7.295608520507812e-05 [0:00:00.000073] [timeout_sampler] 2026-06-11T08:32:28.358685Z [info ] Executing function fixture: snapshot [conftest] 2026-06-11T08:32:28.358852Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-11T08:32:28Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED TEST: TestMLServerModelCar.test_mlserver_model_car_inference[onnx-modelcar-RawDeployment] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-11T08:32:28.583083Z [info ] Delete InferenceService onnx-modelcar [ocp_resources InferenceService] 2026-06-11T08:32:28.615459Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'onnx', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:32:15Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storageUri': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:32:15Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:32:15Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:32:27Z'}], 'name': 'onnx-modelcar', 'namespace': 'onnx-modelcar', 'resourceVersion': '175453036', 'uid': 'd321f272-2ff6-450d-afcd-a979b605c875'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'onnx'}, 'name': '', 'resources': {}, 'runtime': 'mlserver-runtime', 'storageUri': 'oci://quay.io/jooholee/mlserver-onnx@sha256:d0ad00fb6f2caa8f02a0250fc44a576771d0846b2ac8d164ec203b10ec5d604b'}}}, 'status': {'address': {'url': 'http://onnx-modelcar-predictor.onnx-modelcar.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://onnx-modelcar-predictor-onnx-modelcar.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:32:15Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:32:27Z', 'message': 'ReplicaSet "onnx-modelcar-predictor-7587f56fcc" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:32:27Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:32:15Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://onnx-modelcar-predictor.onnx-modelcar.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:32:28.633616Z [info ] Wait until InferenceService onnx-modelcar is deleted [ocp_resources InferenceService] 2026-06-11T08:32:28.633788Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:29.664233Z [info ] Elapsed time: 1.0131871700286865 [0:00:01.013187] [timeout_sampler] 2026-06-11T08:32:29.664426Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-11T08:32:29.695494Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-11T08:32:14Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:32:14Z'}], 'name': 'mlserver-runtime', 'namespace': 'onnx-modelcar', 'resourceVersion': '175452658', 'uid': '2246cbee-d9ce-4ce4-a0a6-83b051860679'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:32:29.712950Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:32:29.713112Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:29.728612Z [info ] Elapsed time: 6.508827209472656e-05 [0:00:00.000065] [timeout_sampler] 2026-06-11T08:32:29.781125Z [info ] Delete Namespace onnx-modelcar [ocp_resources Namespace] 2026-06-11T08:32:29.811754Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'onnx-modelcar', 'uid': '3624eb13-fe41-4376-bd4e-fb7f31523c24', 'resourceVersion': '175452640', 'creationTimestamp': '2026-06-11T08:32:14Z', 'labels': {'kubernetes.io/metadata.name': 'onnx-modelcar', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c34,c4', 'openshift.io/sa.scc.supplemental-groups': '1001130000/10000', 'openshift.io/sa.scc.uid-range': '1001130000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-11T08:32:14Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:32:14Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:32:14Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-11T08:32:29.831620Z [info ] Wait until Namespace onnx-modelcar is deleted [ocp_resources Namespace] 2026-06-11T08:32:29.831791Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] _ 12 of 17 completed, 12 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/mlserver/s3/test_mlserver_s3.py::TestMLServerModels::test_mlserver_model_inference[sklearn-s3-RawDeployment] 2026-06-11T08:32:37.967504Z [info ] Elapsed time: 8.11992335319519 [0:00:08.119923] [timeout_sampler]  ------------------------------- test_mlserver_model_inference[sklearn-s3-RawDeployment] ------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:32:37.968941Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:32:37.969062Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:32:37.969175Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:32:37.969297Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:32:37.969496Z [info ] [cached since 230.5s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/s3/test_mlserver_s3.py::test_mlserver_model_inference[sklearn-s3-RawDeployment]', 1781166757) [sqlalchemy.engine.Engine] 2026-06-11T08:32:37.969636Z [info ] [cached since 230.5s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/s3/test_mlserver_s3.py::test_mlserver_model_inference[sklearn-s3-RawDeployment]', 1781166757) [sqlalchemy.engine.Engine] 2026-06-11T08:32:37.970018Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:32:37.970143Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:32:37.972853Z [info ] Executing session fixture: aws_access_key_id [conftest] 2026-06-11T08:32:37.972970Z [info ] Executing session fixture: aws_secret_access_key [conftest] 2026-06-11T08:32:37.973078Z [info ] Executing session fixture: ci_s3_bucket_endpoint [conftest] 2026-06-11T08:32:37.973180Z [info ] Executing session fixture: valid_aws_config [conftest] 2026-06-11T08:32:38.035535Z [info ] AWS credentials validated successfully via STS GetCallerIdentity [tests.conftest] name=tests.conftest 2026-06-11T08:32:38.038440Z [info ] Executing session fixture: models_s3_bucket_name [conftest] 2026-06-11T08:32:38.038558Z [info ] Executing session fixture: s3_models_storage_uri [conftest] 2026-06-11T08:32:38.038662Z [info ] Executing session fixture: models_s3_bucket_region [conftest] 2026-06-11T08:32:38.038762Z [info ] Executing session fixture: models_s3_bucket_endpoint [conftest] 2026-06-11T08:32:38.038863Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-11T08:32:38.038970Z [info ] Create Namespace sklearn-s3  [ocp_resources Namespace] 2026-06-11T08:32:38.039072Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'sklearn-s3'}, 'spec': {}} [ocp_resources Namespace] 2026-06-11T08:32:38.072048Z [info ] Wait for Namespace sklearn-s3 status to be Active [ocp_resources Namespace] 2026-06-11T08:32:38.072203Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:38.099133Z [info ] Status of Namespace sklearn-s3 is Active [ocp_resources Namespace] 2026-06-11T08:32:38.099286Z [info ] Elapsed time: 3.695487976074219e-05 [0:00:00.000037] [timeout_sampler] 2026-06-11T08:32:38.099959Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-11T08:32:38.100076Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:32:38.179872Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:32:38.180249Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-11T08:32:38.180512Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'sklearn-s3'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:32:38.216989Z [info ] Executing class fixture: kserve_s3_secret [conftest] 2026-06-11T08:32:38.217148Z [info ] Create Secret models-bucket-secret [ocp_resources Secret] 2026-06-11T08:32:38.217319Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'models-bucket-secret', 'annotations': {'serving.kserve.io/s3-endpoint': 's3.us-east-2.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-2', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-verifyssl': '0', 'serving.kserve.io/s3-usehttps': '1'}, 'namespace': 'sklearn-s3'}, 'stringData': '*******'} [ocp_resources Secret] 2026-06-11T08:32:38.249292Z [info ] Wait until Secret models-bucket-secret is created [ocp_resources Secret] 2026-06-11T08:32:38.249515Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:38.262841Z [info ] Elapsed time: 3.743171691894531e-05 [0:00:00.000037] [timeout_sampler] 2026-06-11T08:32:38.263229Z [info ] Executing class fixture: mlserver_model_service_account [conftest] 2026-06-11T08:32:38.263485Z [info ] Create ServiceAccount mlserver-models-bucket-sa [ocp_resources ServiceAccount] 2026-06-11T08:32:38.263594Z [info ] Posting {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'mlserver-models-bucket-sa', 'namespace': 'sklearn-s3'}, 'secrets': [{'name': 'models-bucket-secret'}]} [ocp_resources ServiceAccount] 2026-06-11T08:32:38.291250Z [info ] Executing class fixture: mlserver_inference_service [conftest] 2026-06-11T08:32:38.306093Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:32:38.306504Z [info ] Create InferenceService sklearn [ocp_resources InferenceService] 2026-06-11T08:32:38.306625Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'sklearn', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'sklearn-s3'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'sklearn'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 's3://ods-ci-wisdom/mlserver/model_repository/sklearn/', 'resources': {'requests': {'cpu': '1', 'memory': '2Gi'}, 'limits': {'cpu': '2', 'memory': '4Gi'}}}, 'minReplicas': 1, 'serviceAccountName': 'mlserver-models-bucket-sa'}}} [ocp_resources InferenceService] 2026-06-11T08:32:38.391121Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'mlserver-runtime'}) [timeout_sampler] 2026-06-11T08:32:38.390544Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:32:39.418968Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:32:39.459909Z [info ] Elapsed time: 1.028435468673706 [0:00:01.028435] [timeout_sampler] 2026-06-11T08:32:39.473520Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-06-11T08:32:39.474564Z [info ] Waiting for 899.9999978542328 seconds [0:14:59.999998], retry every 10 seconds. (Function: utilities.infra.get_pods_by_isvc_label Kwargs: {'client': , 'isvc': , 'runtime_name': 'mlserver-runtime'}) [timeout_sampler] 2026-06-11T08:32:49.602857Z [info ] Elapsed time: 10.060962438583374 [0:00:10.060962] [timeout_sampler] 2026-06-11T08:32:49.653245Z [info ] Waiting for 888.7382051944733 seconds [0:14:48.738205], retry every 5 seconds. (Function: ocp_resources.resource.get Kwargs: {'label_selector': 'serving.kserve.io/inferenceservice=sklearn', 'client': , 'namespace': 'sklearn-s3'}) [timeout_sampler] 2026-06-11T08:32:49.653463Z [info ] kind: Deployment api version: apps/v1 [ocp_resources.resource] 2026-06-11T08:32:49.671843Z [info ] Elapsed time: 0.00013375282287597656 [0:00:00.000134] [timeout_sampler] 2026-06-11T08:32:49.671309Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-06-11T08:32:49.730976Z [info ] Waiting for 888.6603062152863 seconds [0:14:48.660306], retry every 5 seconds. (Function: utilities.infra..lambda: deployment.instance) [timeout_sampler] 2026-06-11T08:32:49.752287Z [info ] Elapsed time: 0.00011277198791503906 [0:00:00.000113] [timeout_sampler] 2026-06-11T08:32:49.752489Z [info ] Wait for Deployment sklearn-predictor to be deployed: True [ocp_resources Deployment] 2026-06-11T08:32:49.752627Z [info ] Waiting for 888.639505147934 seconds [0:14:48.639505], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:49.800729Z [info ] Elapsed time: 0.00015616416931152344 [0:00:00.000156] [timeout_sampler] 2026-06-11T08:32:49.800987Z [info ] Waiting for 888.5907633304596 seconds [0:14:48.590763], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-06-11T08:32:49.821426Z [info ] Elapsed time: 4.458427429199219e-05 [0:00:00.000045] [timeout_sampler] 2026-06-11T08:32:49.821596Z [info ] Wait for InferenceService/sklearn's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-11T08:32:49.821730Z [info ] Wait until InferenceService sklearn is created [ocp_resources InferenceService] 2026-06-11T08:32:49.821838Z [info ] Waiting for 888.5698480606079 seconds [0:14:48.569848], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:49.843793Z [info ] Elapsed time: 4.9114227294921875e-05 [0:00:00.000049] [timeout_sampler] 2026-06-11T08:32:49.843951Z [info ] Waiting for 888.547376871109 seconds [0:14:48.547377], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-11T08:32:49.862593Z [info ] Elapsed time: 4.839897155761719e-05 [0:00:00.000048] [timeout_sampler] 2026-06-11T08:32:49.911373Z [info ] Waiting for 888.479856967926 seconds [0:14:48.479857], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-11T08:32:49.927160Z [info ] Elapsed time: 0.00012040138244628906 [0:00:00.000120] [timeout_sampler] 2026-06-11T08:32:50.253421Z [info ] Executing function fixture: mlserver_pod_resource [conftest] 2026-06-11T08:32:50.283281Z [info ] Executing function fixture: snapshot [conftest] 2026-06-11T08:32:50.285155Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] 2026-06-11T08:32:50.285415Z [info ] Executing function fixture: model_format [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-11T08:32:50Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED TEST: TestMLServerModels.test_mlserver_model_inference[sklearn-s3-RawDeployment] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-11T08:32:50.452585Z [info ] Delete InferenceService sklearn [ocp_resources InferenceService] 2026-06-11T08:32:50.485456Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'sklearn', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:32:38Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:minReplicas': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:resources': {'.': {}, 'f:limits': {'.': {}, 'f:cpu': {}, 'f:memory': {}}, 'f:requests': {'.': {}, 'f:cpu': {}, 'f:memory': {}}}, 'f:runtime': {}, 'f:storageUri': {}}, 'f:serviceAccountName': {}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:32:38Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:32:38Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:32:48Z'}], 'name': 'sklearn', 'namespace': 'sklearn-s3', 'resourceVersion': '175453764', 'uid': 'bda9e33c-e447-498d-b797-6c5e4b14b40b'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'minReplicas': 1, 'model': {'modelFormat': {'name': 'sklearn'}, 'name': '', 'resources': {'limits': {'cpu': '2', 'memory': '4Gi'}, 'requests': {'cpu': '1', 'memory': '2Gi'}}, 'runtime': 'mlserver-runtime', 'storageUri': 's3://ods-ci-wisdom/mlserver/model_repository/sklearn/'}, 'serviceAccountName': 'mlserver-models-bucket-sa'}}, 'status': {'address': {'url': 'http://sklearn-predictor.sklearn-s3.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://sklearn-predictor-sklearn-s3.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:32:38Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:32:48Z', 'message': 'ReplicaSet "sklearn-predictor-846cfb85ff" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:32:48Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:32:38Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://sklearn-predictor.sklearn-s3.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:32:50.506708Z [info ] Wait until InferenceService sklearn is deleted [ocp_resources InferenceService] 2026-06-11T08:32:50.506890Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:51.534942Z [info ] Elapsed time: 1.0132930278778076 [0:00:01.013293] [timeout_sampler] 2026-06-11T08:32:51.535122Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-11T08:32:51.603788Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-11T08:32:38Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:32:38Z'}], 'name': 'mlserver-runtime', 'namespace': 'sklearn-s3', 'resourceVersion': '175453418', 'uid': '5267a89e-2b03-427b-b2f8-2779a82b35dc'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:16c8355fb16e3e3579549880a4d16724909014701a99050acb1daf571eca2dfd', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:32:51.621555Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:32:51.621864Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:51.635824Z [info ] Elapsed time: 6.461143493652344e-05 [0:00:00.000065] [timeout_sampler] 2026-06-11T08:32:51.635956Z [info ] Delete ServiceAccount mlserver-models-bucket-sa [ocp_resources ServiceAccount] 2026-06-11T08:32:51.664563Z [info ] Deleting {'kind': 'ServiceAccount', 'apiVersion': 'v1', 'metadata': {'name': 'mlserver-models-bucket-sa', 'namespace': 'sklearn-s3', 'uid': '6d68c24b-9fe7-4d67-9d5c-bdf25ca7abc8', 'resourceVersion': '175453426', 'creationTimestamp': '2026-06-11T08:32:38Z', 'annotations': {'openshift.io/internal-registry-pull-secret-ref': 'mlserver-models-bucket-sa-dockercfg-jl8mt'}, 'managedFields': [{'manager': 'openshift.io/image-registry-pull-secrets_service-account-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-11T08:32:38Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:imagePullSecrets': {}, 'f:metadata': {'f:annotations': {'f:openshift.io/internal-registry-pull-secret-ref': {}}}, 'f:secrets': {'k:{"name":"mlserver-models-bucket-sa-dockercfg-jl8mt"}': {}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:32:38Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:secrets': {'.': {}, 'k:{"name":"models-bucket-secret"}': {}}}}]}, 'secrets': [{'name': 'models-bucket-secret'}, {'name': 'mlserver-models-bucket-sa-dockercfg-jl8mt'}], 'imagePullSecrets': [{'name': 'mlserver-models-bucket-sa-dockercfg-jl8mt'}]} [ocp_resources ServiceAccount] 2026-06-11T08:32:51.680574Z [info ] Wait until ServiceAccount mlserver-models-bucket-sa is deleted [ocp_resources ServiceAccount] 2026-06-11T08:32:51.680860Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:51.694930Z [info ] Elapsed time: 3.457069396972656e-05 [0:00:00.000035] [timeout_sampler] 2026-06-11T08:32:51.695325Z [info ] Delete Secret models-bucket-secret [ocp_resources Secret] 2026-06-11T08:32:51.734267Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'models-bucket-secret', 'namespace': 'sklearn-s3', 'uid': 'f9bc1bf7-38c9-4a50-a993-baf6ae3ec5b8', 'resourceVersion': '175453421', 'creationTimestamp': '2026-06-11T08:32:38Z', 'annotations': {'serving.kserve.io/s3-endpoint': 's3.us-east-2.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-2', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-usehttps': '1', 'serving.kserve.io/s3-verifyssl': '0'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:32:38Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:AWS_ACCESS_KEY_ID': {}, 'f:AWS_SECRET_ACCESS_KEY': {}}, 'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/s3-endpoint': {}, 'f:serving.kserve.io/s3-region': {}, 'f:serving.kserve.io/s3-useanoncredential': {}, 'f:serving.kserve.io/s3-usehttps': {}, 'f:serving.kserve.io/s3-verifyssl': {}}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-06-11T08:32:51.785146Z [info ] Wait until Secret models-bucket-secret is deleted [ocp_resources Secret] 2026-06-11T08:32:51.785514Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:32:51.798953Z [info ] Elapsed time: 4.267692565917969e-05 [0:00:00.000043] [timeout_sampler] 2026-06-11T08:32:51.852142Z [info ] Delete Namespace sklearn-s3  [ocp_resources Namespace] 2026-06-11T08:32:51.881689Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'sklearn-s3', 'uid': '991a0233-e4e9-47f2-94f2-c003272c5256', 'resourceVersion': '175453404', 'creationTimestamp': '2026-06-11T08:32:38Z', 'labels': {'kubernetes.io/metadata.name': 'sklearn-s3', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c34,c9', 'openshift.io/sa.scc.supplemental-groups': '1001140000/10000', 'openshift.io/sa.scc.uid-range': '1001140000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-11T08:32:38Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:32:38Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:32:38Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-11T08:32:51.898910Z [info ] Wait until Namespace sklearn-s3 is deleted [ocp_resources Namespace] 2026-06-11T08:32:51.899047Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] _ 13 of 17 completed, 13 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::TestOpenVINOModels::test_openvino_model_inference[onnx-raw-rest-deployment] 2026-06-11T08:33:00.065743Z [info ] Elapsed time: 8.151373863220215 [0:00:08.151374] [timeout_sampler]  ------------------------------- test_openvino_model_inference[onnx-raw-rest-deployment] ------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:33:00.067291Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:33:00.067896Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:33:00.068484Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:33:00.069855Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:33:00.070112Z [info ] [cached since 252.6s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::test_openvino_model_inference[onnx-raw-rest-deployment]', 1781166780) [sqlalchemy.engine.Engine] 2026-06-11T08:33:00.070256Z [info ] [cached since 252.6s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::test_openvino_model_inference[onnx-raw-rest-deployment]', 1781166780) [sqlalchemy.engine.Engine] 2026-06-11T08:33:00.070402Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:33:00.070516Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:33:00.070625Z [info ] Executing session fixture: s3_models_storage_uri [conftest] 2026-06-11T08:33:00.070732Z [info ] Executing session fixture: gpu_count_on_cluster [conftest] 2026-06-11T08:33:00.215718Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-11T08:33:00.216094Z [info ] Create Namespace onnx-raw-rest [ocp_resources Namespace] 2026-06-11T08:33:00.216225Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'onnx-raw-rest'}, 'spec': {}} [ocp_resources Namespace] 2026-06-11T08:33:00.267628Z [info ] Wait for Namespace onnx-raw-rest status to be Active [ocp_resources Namespace] 2026-06-11T08:33:00.267877Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-11T08:33:00.289700Z [info ] Status of Namespace onnx-raw-rest is Active [ocp_resources Namespace] 2026-06-11T08:33:00.289983Z [info ] Elapsed time: 4.6253204345703125e-05 [0:00:00.000046] [timeout_sampler] 2026-06-11T08:33:00.290173Z [info ] Executing class fixture: openvino_serving_runtime [conftest] 2026-06-11T08:33:00.290278Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:33:00.335663Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:33:00.335942Z [info ] Create ServingRuntime openvino-runtime [ocp_resources ServingRuntime] 2026-06-11T08:33:00.336164Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'openvino-runtime', 'namespace': 'onnx-raw-rest'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:a7a965959765c52a705ac632d19d823e4ad843ad3657e395d708da7e5401a29f', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:33:00.374679Z [info ] Executing class fixture: kserve_s3_secret [conftest] 2026-06-11T08:33:00.375031Z [info ] Create Secret models-bucket-secret [ocp_resources Secret] 2026-06-11T08:33:00.375761Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'models-bucket-secret', 'annotations': {'serving.kserve.io/s3-endpoint': 's3.us-east-2.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-2', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-verifyssl': '0', 'serving.kserve.io/s3-usehttps': '1'}, 'namespace': 'onnx-raw-rest'}, 'stringData': '*******'} [ocp_resources Secret] 2026-06-11T08:33:00.435222Z [info ] Wait until Secret models-bucket-secret is created [ocp_resources Secret] 2026-06-11T08:33:00.435389Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-11T08:33:00.448613Z [info ] Elapsed time: 8.058547973632812e-05 [0:00:00.000081] [timeout_sampler] 2026-06-11T08:33:00.448988Z [info ] Executing class fixture: openvino_model_service_account [conftest] 2026-06-11T08:33:00.449183Z [info ] Create ServiceAccount openvino-models-bucket-sa [ocp_resources ServiceAccount] 2026-06-11T08:33:00.449293Z [info ] Posting {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'openvino-models-bucket-sa', 'namespace': 'onnx-raw-rest'}, 'secrets': [{'name': 'models-bucket-secret'}]} [ocp_resources ServiceAccount] 2026-06-11T08:33:00.703727Z [info ] Executing class fixture: openvino_inference_service [conftest] 2026-06-11T08:33:00.746615Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:33:00.747028Z [info ] Create InferenceService onnx  [ocp_resources InferenceService] 2026-06-11T08:33:00.747223Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'onnx', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'onnx-raw-rest'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'openvino_ir'}, 'version': None, 'runtime': 'openvino-runtime', 'storageUri': 's3://ods-ci-wisdom/openvino/model_repository/onnx/', 'resources': {'requests': {'cpu': '1', 'memory': '2Gi'}, 'limits': {'cpu': '2', 'memory': '4Gi'}}}, 'minReplicas': 1, 'serviceAccountName': 'openvino-models-bucket-sa'}}} [ocp_resources InferenceService] 2026-06-11T08:33:00.791614Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'openvino-runtime'}) [timeout_sampler] 2026-06-11T08:33:00.790737Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:33:01.817274Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:33:01.846700Z [info ] Elapsed time: 1.0265836715698242 [0:00:01.026584] [timeout_sampler] 2026-06-11T08:33:01.860148Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-06-11T08:33:01.861206Z [info ] Waiting for 899.999997138977 seconds [0:14:59.999997], retry every 10 seconds. (Function: utilities.infra.get_pods_by_isvc_label Kwargs: {'client': , 'isvc': , 'runtime_name': 'openvino-runtime'}) [timeout_sampler] 2026-06-11T08:33:11.979632Z [info ] Elapsed time: 10.062697887420654 [0:00:10.062698] [timeout_sampler] 2026-06-11T08:33:11.992511Z [info ] Waiting for 888.798798084259 seconds [0:14:48.798798], retry every 5 seconds. (Function: ocp_resources.resource.get Kwargs: {'label_selector': 'serving.kserve.io/inferenceservice=onnx', 'client': , 'namespace': 'onnx-raw-rest'}) [timeout_sampler] 2026-06-11T08:33:12.005817Z [info ] Elapsed time: 8.654594421386719e-05 [0:00:00.000087] [timeout_sampler] 2026-06-11T08:33:12.005115Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-06-11T08:33:12.052812Z [info ] Waiting for 888.7386856079102 seconds [0:14:48.738686], retry every 5 seconds. (Function: utilities.infra..lambda: deployment.instance) [timeout_sampler] 2026-06-11T08:33:12.067086Z [info ] Elapsed time: 0.0001285076141357422 [0:00:00.000129] [timeout_sampler] 2026-06-11T08:33:12.067256Z [info ] Wait for Deployment onnx-predictor to be deployed: True [ocp_resources Deployment] 2026-06-11T08:33:12.067490Z [info ] Waiting for 888.724029302597 seconds [0:14:48.724029], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-06-11T08:33:12.081053Z [info ] Elapsed time: 2.956390380859375e-05 [0:00:00.000030] [timeout_sampler] 2026-06-11T08:33:12.081216Z [info ] Waiting for 888.710072517395 seconds [0:14:48.710073], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-06-11T08:33:12.095055Z [info ] Elapsed time: 2.9802322387695312e-05 [0:00:00.000030] [timeout_sampler] 2026-06-11T08:33:12.095224Z [info ] Wait for InferenceService/onnx's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-11T08:33:12.095385Z [info ] Wait until InferenceService onnx is created [ocp_resources InferenceService] 2026-06-11T08:33:12.095507Z [info ] Waiting for 888.6961486339569 seconds [0:14:48.696149], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-11T08:33:12.110270Z [info ] Elapsed time: 2.7418136596679688e-05 [0:00:00.000027] [timeout_sampler] 2026-06-11T08:33:12.110531Z [info ] Waiting for 888.680882692337 seconds [0:14:48.680883], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-11T08:33:12.124020Z [info ] Elapsed time: 3.600120544433594e-05 [0:00:00.000036] [timeout_sampler] 2026-06-11T08:33:12.139040Z [info ] Waiting for 888.6520779132843 seconds [0:14:48.652078], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-11T08:33:12.152617Z [info ] Elapsed time: 4.506111145019531e-05 [0:00:00.000045] [timeout_sampler] 2026-06-11T08:33:12.171260Z [info ] Executing class fixture: protocol [conftest] 2026-06-11T08:33:12.171533Z [info ] Executing function fixture: openvino_pod_resource [conftest] 2026-06-11T08:33:12.208038Z [info ] Executing function fixture: snapshot [conftest] 2026-06-11T08:33:12.208664Z [info ] Executing function fixture: openvino_response_snapshot [conftest] 2026-06-11T08:33:12.210008Z [info ] Executing function fixture: model_format [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- FAILED TEST: TestOpenVINOModels.test_openvino_model_inference[onnx-raw-rest-deployment] STATUS: FAILED 2026-06-11T08:33:12.250913Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 179, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1720, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 166, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py", line 149, in test_openvino_model_inference validate_inference_request( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ pod_name=openvino_pod_resource.name, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... model_version=model_version, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/utils.py", line 113, in validate_inference_request response = run_openvino_inference( pod_name=pod_name, ...<2 lines>... model_version=model_version, ) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/utils.py", line 87, in run_openvino_inference raise ValueError(f"Invalid deployment_mode {deployment_mode}") ValueError: Invalid deployment_mode Standard [conftest] 2026-06-11T08:33:12.255469Z [info ] Must-gather collection is enabled for /home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::test_openvino_model_inference[onnx-raw-rest-deployment]. [conftest] 2026-06-11T08:33:12.255717Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:33:12.255870Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:33:12.256023Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-11T08:33:12.256174Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-11T08:33:12.256394Z [info ] [generated in 0.00019s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::test_openvino_model_inference[onnx-raw-rest-deployment]', 1, 0) [sqlalchemy.engine.Engine] 2026-06-11T08:33:12.261826Z [info ] [generated in 0.00019s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py::test_openvino_model_inference[onnx-raw-rest-deployment]', 1, 0) [sqlalchemy.engine.Engine] 2026-06-11T08:33:12.261982Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-11T08:33:12.262097Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-11T08:33:12.262215Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-11T08:33:12.262325Z [info ] kind: Subscription api version: operators.coreos.com/v1alpha1 [ocp_resources.resource] 2026-06-11T08:33:12.422723Z [info ] Checking subscription rhods-operator [utilities.infra] name=utilities.infra 2026-06-11T08:33:12.436682Z [info ] Expected CSV: rhods-operator.3.4.1 [utilities.infra] name=utilities.infra 2026-06-11T08:33:12.439083Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-11T08:33:12.583134Z [info ] Running oc adm must-gather --dest-dir=/home/odh/opendatahub-tests/results/must-gather-collected/model_serving/model_runtime/openvino/test_ovms_model_deployment/TestOpenVINOModels/test_openvino_model_inference_onnx-raw-rest-deployment_/pytest_exception_interact --since=300s --image=registry.redhat.io/rhoai/odh-must-gather-rhel9@sha256:ec1d4bcb2ba86e7b252cbc95d7eb16e4f468520d76d461f8adf4036746051b15 command [pyhelper_utils.shell] 2026-06-11T08:38:38.768395Z [info ] /home/odh/opendatahub-tests/results/must-gather-collected/model_serving/model_runtime/openvino/test_ovms_model_deployment/TestOpenVINOModels/test_openvino_model_inference_onnx-raw-rest-deployment_/pytest_exception_interact/mg-1781166780.zip is collected successfully [utilities.must_gather_collector] name=utilities.must_gather_collector ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-11T08:38:38.772663Z [info ] Delete InferenceService onnx  [ocp_resources InferenceService] 2026-06-11T08:38:38.857129Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'openvino_ir', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:33:00Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:minReplicas': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:resources': {'.': {}, 'f:limits': {'.': {}, 'f:cpu': {}, 'f:memory': {}}, 'f:requests': {'.': {}, 'f:cpu': {}, 'f:memory': {}}}, 'f:runtime': {}, 'f:storageUri': {}}, 'f:serviceAccountName': {}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:33:00Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:33:00Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:33:04Z'}], 'name': 'onnx', 'namespace': 'onnx-raw-rest', 'resourceVersion': '175454366', 'uid': '39443a4f-7e55-4b74-af00-3646b29249fd'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'minReplicas': 1, 'model': {'modelFormat': {'name': 'openvino_ir'}, 'name': '', 'resources': {'limits': {'cpu': '2', 'memory': '4Gi'}, 'requests': {'cpu': '1', 'memory': '2Gi'}}, 'runtime': 'openvino-runtime', 'storageUri': 's3://ods-ci-wisdom/openvino/model_repository/onnx/'}, 'serviceAccountName': 'openvino-models-bucket-sa'}}, 'status': {'address': {'url': 'http://onnx-predictor.onnx-raw-rest.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://onnx-predictor-onnx-raw-rest.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:33:00Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:33:04Z', 'message': 'ReplicaSet "onnx-predictor-9499b584b" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:33:04Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:33:00Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'openvino-runtime', 'url': 'http://onnx-predictor.onnx-raw-rest.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:38:38.878568Z [info ] Wait until InferenceService onnx is deleted [ocp_resources InferenceService] 2026-06-11T08:38:38.878721Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:38:39.920466Z [info ] Elapsed time: 1.026125431060791 [0:00:01.026125] [timeout_sampler] 2026-06-11T08:38:39.920652Z [info ] Delete ServingRuntime openvino-runtime [ocp_resources ServingRuntime] 2026-06-11T08:38:39.950465Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'creationTimestamp': '2026-06-11T08:33:00Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/recommended-accelerators': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:33:00Z'}], 'name': 'openvino-runtime', 'namespace': 'onnx-raw-rest', 'resourceVersion': '175454156', 'uid': '910daa93-7ccb-4c1b-b7c2-4c4b79008911'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:a7a965959765c52a705ac632d19d823e4ad843ad3657e395d708da7e5401a29f', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-06-11T08:38:39.967670Z [info ] Wait until ServingRuntime openvino-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:38:39.967840Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:38:39.983190Z [info ] Elapsed time: 5.6743621826171875e-05 [0:00:00.000057] [timeout_sampler] 2026-06-11T08:38:39.983380Z [info ] Delete ServiceAccount openvino-models-bucket-sa [ocp_resources ServiceAccount] 2026-06-11T08:38:40.011452Z [info ] Deleting {'kind': 'ServiceAccount', 'apiVersion': 'v1', 'metadata': {'name': 'openvino-models-bucket-sa', 'namespace': 'onnx-raw-rest', 'uid': 'f94eb3c0-3ed4-4538-a5fc-aa7610fe3c23', 'resourceVersion': '175454173', 'creationTimestamp': '2026-06-11T08:33:00Z', 'annotations': {'openshift.io/internal-registry-pull-secret-ref': 'openvino-models-bucket-sa-dockercfg-wsqvk'}, 'managedFields': [{'manager': 'openshift.io/image-registry-pull-secrets_service-account-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-11T08:33:00Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:imagePullSecrets': {}, 'f:metadata': {'f:annotations': {'f:openshift.io/internal-registry-pull-secret-ref': {}}}, 'f:secrets': {'k:{"name":"openvino-models-bucket-sa-dockercfg-wsqvk"}': {}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:33:00Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:secrets': {'.': {}, 'k:{"name":"models-bucket-secret"}': {}}}}]}, 'secrets': [{'name': 'models-bucket-secret'}, {'name': 'openvino-models-bucket-sa-dockercfg-wsqvk'}], 'imagePullSecrets': [{'name': 'openvino-models-bucket-sa-dockercfg-wsqvk'}]} [ocp_resources ServiceAccount] 2026-06-11T08:38:40.063663Z [info ] Wait until ServiceAccount openvino-models-bucket-sa is deleted [ocp_resources ServiceAccount] 2026-06-11T08:38:40.063857Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:38:40.078857Z [info ] Elapsed time: 8.130073547363281e-05 [0:00:00.000081] [timeout_sampler] 2026-06-11T08:38:40.079173Z [info ] Delete Secret models-bucket-secret [ocp_resources Secret] 2026-06-11T08:38:40.138506Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'models-bucket-secret', 'namespace': 'onnx-raw-rest', 'uid': '12221649-0010-4b89-8923-58cd9f0c2390', 'resourceVersion': '175454161', 'creationTimestamp': '2026-06-11T08:33:00Z', 'annotations': {'serving.kserve.io/s3-endpoint': 's3.us-east-2.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-2', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-usehttps': '1', 'serving.kserve.io/s3-verifyssl': '0'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:33:00Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:AWS_ACCESS_KEY_ID': {}, 'f:AWS_SECRET_ACCESS_KEY': {}}, 'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/s3-endpoint': {}, 'f:serving.kserve.io/s3-region': {}, 'f:serving.kserve.io/s3-useanoncredential': {}, 'f:serving.kserve.io/s3-usehttps': {}, 'f:serving.kserve.io/s3-verifyssl': {}}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-06-11T08:38:40.155169Z [info ] Wait until Secret models-bucket-secret is deleted [ocp_resources Secret] 2026-06-11T08:38:40.155406Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:38:40.170487Z [info ] Elapsed time: 4.076957702636719e-05 [0:00:00.000041] [timeout_sampler] 2026-06-11T08:38:40.186447Z [info ] Delete Namespace onnx-raw-rest [ocp_resources Namespace] 2026-06-11T08:38:40.254447Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'onnx-raw-rest', 'uid': 'e6a75ead-49e5-4bcf-910b-cb5c16e0177f', 'resourceVersion': '175454143', 'creationTimestamp': '2026-06-11T08:33:00Z', 'labels': {'kubernetes.io/metadata.name': 'onnx-raw-rest', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c34,c14', 'openshift.io/sa.scc.supplemental-groups': '1001150000/10000', 'openshift.io/sa.scc.uid-range': '1001150000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-11T08:33:00Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:33:00Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:33:00Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-11T08:38:40.271584Z [info ] Wait until Namespace onnx-raw-rest is deleted [ocp_resources Namespace] 2026-06-11T08:38:40.271819Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] __ TestOpenVINOModels.test_openvino_model_inference[onnx-raw-rest-deployment] __ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 179, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1720, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 166, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_model_deployment.py", line 149, in test_openvino_model_inference validate_inference_request( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ pod_name=openvino_pod_resource.name, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... model_version=model_version, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/utils.py", line 113, in validate_inference_request response = run_openvino_inference( pod_name=pod_name, ...<2 lines>... model_version=model_version, ) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/utils.py", line 87, in run_openvino_inference raise ValueError(f"Invalid deployment_mode {deployment_mode}") ValueError: Invalid deployment_mode Standard _ 14 of 17 completed, 13 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/openvino/test_ovms_smoke.py::TestOVMSSmokeInOpenShift::test_ovms_smoke_runs_in_openshift[ovms-smoke] 2026-06-11T08:38:48.433318Z [info ] Elapsed time: 8.144631385803223 [0:00:08.144631] [timeout_sampler]  ------------------------------------ test_ovms_smoke_runs_in_openshift[ovms-smoke] ------------------------------------ -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:38:48.436589Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:38:48.438784Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:38:48.439128Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:38:48.439292Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:38:48.439622Z [info ] [cached since 600.9s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_smoke.py::test_ovms_smoke_runs_in_openshift[ovms-smoke]', 1781167128) [sqlalchemy.engine.Engine] 2026-06-11T08:38:48.439831Z [info ] [cached since 600.9s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/openvino/test_ovms_smoke.py::test_ovms_smoke_runs_in_openshift[ovms-smoke]', 1781167128) [sqlalchemy.engine.Engine] 2026-06-11T08:38:48.439960Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:38:48.440069Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:38:48.440173Z [info ] Executing session fixture: ovms_runtime_image [conftest] 2026-06-11T08:38:48.440278Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:38:48.513217Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-11T08:38:48.513617Z [info ] Create Namespace ovms-smoke  [ocp_resources Namespace] 2026-06-11T08:38:48.513947Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'ovms-smoke'}, 'spec': {}} [ocp_resources Namespace] 2026-06-11T08:38:48.558979Z [info ] Wait for Namespace ovms-smoke status to be Active [ocp_resources Namespace] 2026-06-11T08:38:48.559267Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-11T08:38:48.593033Z [info ] Status of Namespace ovms-smoke is Active [ocp_resources Namespace] 2026-06-11T08:38:48.593191Z [info ] Elapsed time: 4.553794860839844e-05 [0:00:00.000046] [timeout_sampler] 2026-06-11T08:38:48.593334Z [info ] Executing class fixture: ovms_smoke_scripts_configmap [conftest] 2026-06-11T08:38:48.595774Z [info ] Create ConfigMap ovms-smoke-scripts [ocp_resources ConfigMap] 2026-06-11T08:38:48.596010Z [info ] Posting {'apiVersion': 'v1', 'kind': 'ConfigMap', 'metadata': {'name': 'ovms-smoke-scripts', 'namespace': 'ovms-smoke'}, 'data': '*******'} [ocp_resources ConfigMap] 2026-06-11T08:38:48.643626Z [info ] Executing class fixture: ovms_smoke_pod [conftest] 2026-06-11T08:38:48.643949Z [info ] Create Pod ovms-smoke-pod  [ocp_resources Pod] 2026-06-11T08:38:48.644079Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'ovms-smoke-pod', 'namespace': 'ovms-smoke'}, 'spec': {'containers': [{'name': 'ovms-smoke', 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:a7a965959765c52a705ac632d19d823e4ad843ad3657e395d708da7e5401a29f', 'command': ['/bin/sh', '-c'], 'args': ['python3 /scripts/ovms_smoketest.py && python3 /scripts/smoke.py'], 'env': [{'name': 'HOME', 'value': '/tmp'}, {'name': 'HF_HOME', 'value': '/tmp/hf_cache'}, {'name': 'MPLCONFIGDIR', 'value': '/tmp/matplotlib'}], 'volumeMounts': [{'name': 'smoke-scripts', 'mountPath': '/scripts', 'readOnly': True}], 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'runAsNonRoot': True, 'seccompProfile': {'type': 'RuntimeDefault'}}}], 'restartPolicy': 'Never', 'volumes': [{'name': 'smoke-scripts', 'configMap': {'name': 'ovms-smoke-scripts'}}]}} [ocp_resources Pod] 2026-06-11T08:38:48.710437Z [info ] Waiting for OVMS smoke Pod to complete [tests.model_serving.model_runtime.openvino.conftest] name=tests.model_serving.model_runtime.openvino.conftest 2026-06-11T08:38:48.711796Z [info ] Wait for Pod ovms-smoke-pod status to be Succeeded [ocp_resources Pod] 2026-06-11T08:38:48.711978Z [info ] Waiting for 300 seconds [0:05:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-11T08:38:48.724827Z [info ] Status of Pod ovms-smoke-pod is Pending [ocp_resources Pod] 2026-06-11T08:38:49.748779Z [info ] Status of Pod ovms-smoke-pod is Running [ocp_resources Pod] 2026-06-11T08:39:25.595305Z [info ] Status of Pod ovms-smoke-pod is Succeeded [ocp_resources Pod] 2026-06-11T08:39:25.595542Z [info ] Elapsed time: 36.8672616481781 [0:00:36.867262] [timeout_sampler] --------------------------------------------------------- CALL --------------------------------------------------------- PASSED TEST: TestOVMSSmokeInOpenShift.test_ovms_smoke_runs_in_openshift[ovms-smoke] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-11T08:39:25.615533Z [info ] Delete Pod ovms-smoke-pod  [ocp_resources Pod] 2026-06-11T08:39:25.664090Z [info ] Deleting {'kind': 'Pod', 'apiVersion': 'v1', 'metadata': {'name': 'ovms-smoke-pod', 'namespace': 'ovms-smoke', 'uid': '35fae79e-e835-4d48-90eb-15898b214d9a', 'resourceVersion': '175464511', 'generation': 1, 'creationTimestamp': '2026-06-11T08:38:48Z', 'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.128.2.245/23"],"mac_address":"0a:58:0a:80:02:f5","gateway_ips":["10.128.2.1"],"routes":[{"dest":"10.128.0.0/14","nextHop":"10.128.2.1"},{"dest":"172.30.0.0/16","nextHop":"10.128.2.1"},{"dest":"169.254.0.5/32","nextHop":"10.128.2.1"},{"dest":"100.64.0.0/16","nextHop":"10.128.2.1"}],"ip_address":"10.128.2.245/23","gateway_ip":"10.128.2.1","role":"primary"}}', 'k8s.v1.cni.cncf.io/network-status': '[{\n "name": "ovn-kubernetes",\n "interface": "eth0",\n "ips": [\n "10.128.2.245"\n ],\n "mac": "0a:58:0a:80:02:f5",\n "default": true,\n "dns": {}\n}]', 'openshift.io/scc': 'openshift-ai-llminferenceservice-multi-node-scc', 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default', 'security.openshift.io/validated-scc-subject-type': 'user'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:38:48Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:containers': {'k:{"name":"ovms-smoke"}': {'.': {}, 'f:args': {}, 'f:command': {}, 'f:env': {'.': {}, 'k:{"name":"HF_HOME"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"HOME"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"MPLCONFIGDIR"}': {'.': {}, 'f:name': {}, 'f:value': {}}}, 'f:image': {}, 'f:imagePullPolicy': {}, 'f:name': {}, 'f:resources': {}, 'f:securityContext': {'.': {}, 'f:allowPrivilegeEscalation': {}, 'f:capabilities': {'.': {}, 'f:drop': {}}, 'f:runAsNonRoot': {}, 'f:seccompProfile': {'.': {}, 'f:type': {}}}, 'f:terminationMessagePath': {}, 'f:terminationMessagePolicy': {}, 'f:volumeMounts': {'.': {}, 'k:{"mountPath":"/scripts"}': {'.': {}, 'f:mountPath': {}, 'f:name': {}, 'f:readOnly': {}}}}}, 'f:dnsPolicy': {}, 'f:enableServiceLinks': {}, 'f:restartPolicy': {}, 'f:schedulerName': {}, 'f:securityContext': {}, 'f:terminationGracePeriodSeconds': {}, 'f:volumes': {'.': {}, 'k:{"name":"smoke-scripts"}': {'.': {}, 'f:configMap': {'.': {}, 'f:defaultMode': {}, 'f:name': {}}, 'f:name': {}}}}}}, {'manager': 'ods-qe-psi-23-v5877-worker-0-4m76g', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:38:48Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}}, 'subresource': 'status'}, {'manager': 'multus-daemon', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:38:49Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}}, 'subresource': 'status'}, {'manager': 'kubelet', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:39:24Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {}, 'f:lastProbeTime': {}, 'f:lastTransitionTime': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"Initialized"}': {'.': {}, 'f:lastProbeTime': {}, 'f:lastTransitionTime': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"PodReadyToStartContainers"}': {'.': {}, 'f:lastProbeTime': {}, 'f:lastTransitionTime': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"Ready"}': {'.': {}, 'f:lastProbeTime': {}, 'f:lastTransitionTime': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}}, 'f:containerStatuses': {}, 'f:hostIP': {}, 'f:hostIPs': {}, 'f:phase': {}, 'f:podIP': {}, 'f:podIPs': {'.': {}, 'k:{"ip":"10.128.2.245"}': {'.': {}, 'f:ip': {}}}, 'f:startTime': {}}}, 'subresource': 'status'}]}, 'spec': {'volumes': [{'name': 'smoke-scripts', 'configMap': {'name': 'ovms-smoke-scripts', 'defaultMode': 420}}, {'name': 'kube-api-access-l5vm8', 'projected': {'sources': [{'serviceAccountToken': {'expirationSeconds': 3607, 'path': 'token'}}, {'configMap': {'name': 'kube-root-ca.crt', 'items': [{'key': 'ca.crt', 'path': 'ca.crt'}]}}, {'downwardAPI': {'items': [{'path': 'namespace', 'fieldRef': {'apiVersion': 'v1', 'fieldPath': 'metadata.namespace'}}]}}, {'configMap': {'name': 'openshift-service-ca.crt', 'items': [{'key': 'service-ca.crt', 'path': 'service-ca.crt'}]}}], 'defaultMode': 420}}], 'containers': [{'name': 'ovms-smoke', 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:a7a965959765c52a705ac632d19d823e4ad843ad3657e395d708da7e5401a29f', 'command': ['/bin/sh', '-c'], 'args': ['python3 /scripts/ovms_smoketest.py && python3 /scripts/smoke.py'], 'env': [{'name': 'HOME', 'value': '/tmp'}, {'name': 'HF_HOME', 'value': '/tmp/hf_cache'}, {'name': 'MPLCONFIGDIR', 'value': '/tmp/matplotlib'}], 'resources': {}, 'volumeMounts': [{'name': 'smoke-scripts', 'readOnly': True, 'mountPath': '/scripts'}, {'name': 'kube-api-access-l5vm8', 'readOnly': True, 'mountPath': '/var/run/secrets/kubernetes.io/serviceaccount'}], 'terminationMessagePath': '/dev/termination-log', 'terminationMessagePolicy': 'File', 'imagePullPolicy': 'IfNotPresent', 'securityContext': {'capabilities': {'drop': ['ALL']}, 'runAsUser': 1001170000, 'runAsNonRoot': True, 'allowPrivilegeEscalation': False, 'seccompProfile': {'type': 'RuntimeDefault'}}}], 'restartPolicy': 'Never', 'terminationGracePeriodSeconds': 30, 'dnsPolicy': 'ClusterFirst', 'serviceAccountName': 'default', 'serviceAccount': 'default', 'nodeName': 'ods-qe-psi-23-v5877-worker-0-4m76g', 'securityContext': {'seLinuxOptions': {'level': 's0:c34,c24'}, 'seccompProfile': {'type': 'RuntimeDefault'}}, 'imagePullSecrets': [{'name': 'default-dockercfg-f2kl7'}], 'schedulerName': 'default-scheduler', 'tolerations': [{'key': 'node.kubernetes.io/not-ready', 'operator': 'Exists', 'effect': 'NoExecute', 'tolerationSeconds': 300}, {'key': 'node.kubernetes.io/unreachable', 'operator': 'Exists', 'effect': 'NoExecute', 'tolerationSeconds': 300}], 'priority': 0, 'enableServiceLinks': True, 'preemptionPolicy': 'PreemptLowerPriority'}, 'status': {'phase': 'Succeeded', 'conditions': [{'type': 'PodReadyToStartContainers', 'status': 'False', 'lastProbeTime': None, 'lastTransitionTime': '2026-06-11T08:39:24Z'}, {'type': 'Initialized', 'status': 'True', 'lastProbeTime': None, 'lastTransitionTime': '2026-06-11T08:38:48Z', 'reason': 'PodCompleted'}, {'type': 'Ready', 'status': 'False', 'lastProbeTime': None, 'lastTransitionTime': '2026-06-11T08:39:23Z', 'reason': 'PodCompleted'}, {'type': 'ContainersReady', 'status': 'False', 'lastProbeTime': None, 'lastTransitionTime': '2026-06-11T08:39:23Z', 'reason': 'PodCompleted'}, {'type': 'PodScheduled', 'status': 'True', 'lastProbeTime': None, 'lastTransitionTime': '2026-06-11T08:38:48Z'}], 'hostIP': '192.169.0.196', 'hostIPs': [{'ip': '192.169.0.196'}], 'podIP': '10.128.2.245', 'podIPs': [{'ip': '10.128.2.245'}], 'startTime': '2026-06-11T08:38:48Z', 'containerStatuses': [{'name': 'ovms-smoke', 'state': {'terminated': {'exitCode': 0, 'reason': 'Completed', 'startedAt': '2026-06-11T08:38:49Z', 'finishedAt': '2026-06-11T08:39:23Z', 'containerID': 'cri-o://a6ab510ce90d7f5d9b5458025d6287c40de824c16d30f04071ddfec163e6a4e5'}}, 'lastState': {}, 'ready': False, 'restartCount': 0, 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:a7a965959765c52a705ac632d19d823e4ad843ad3657e395d708da7e5401a29f', 'imageID': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:837a40939b80dc32d6b8ccff555699c4ec4ea0302f43a4dc1ce7c781b86196a9', 'containerID': 'cri-o://a6ab510ce90d7f5d9b5458025d6287c40de824c16d30f04071ddfec163e6a4e5', 'started': False, 'resources': {}, 'volumeMounts': [{'name': 'smoke-scripts', 'mountPath': '/scripts', 'readOnly': True, 'recursiveReadOnly': 'Disabled'}, {'name': 'kube-api-access-l5vm8', 'mountPath': '/var/run/secrets/kubernetes.io/serviceaccount', 'readOnly': True, 'recursiveReadOnly': 'Disabled'}], 'user': {'linux': {'uid': 1001170000, 'gid': 0, 'supplementalGroups': [0]}}}], 'qosClass': 'BestEffort'}} [ocp_resources Pod] 2026-06-11T08:39:25.695768Z [info ] Wait until Pod ovms-smoke-pod is deleted [ocp_resources Pod] 2026-06-11T08:39:25.695960Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:39:25.710566Z [info ] Elapsed time: 5.221366882324219e-05 [0:00:00.000052] [timeout_sampler] 2026-06-11T08:39:25.710753Z [info ] Delete ConfigMap ovms-smoke-scripts [ocp_resources ConfigMap] 2026-06-11T08:39:25.742832Z [info ] Deleting {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'ovms-smoke-scripts', 'namespace': 'ovms-smoke', 'uid': '8b63ef94-2efe-4900-8355-d142f34d9bb3', 'resourceVersion': '175463582', 'creationTimestamp': '2026-06-11T08:38:48Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:38:48Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:ovms_smoketest.py': {}, 'f:smoke.py': {}}}}]}, 'data': '*******'} [ocp_resources ConfigMap] 2026-06-11T08:39:25.763738Z [info ] Wait until ConfigMap ovms-smoke-scripts is deleted [ocp_resources ConfigMap] 2026-06-11T08:39:25.763958Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:39:25.778097Z [info ] Elapsed time: 4.363059997558594e-05 [0:00:00.000044] [timeout_sampler] 2026-06-11T08:39:25.799543Z [info ] Delete Namespace ovms-smoke  [ocp_resources Namespace] 2026-06-11T08:39:25.831712Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'ovms-smoke', 'uid': 'f18b7ca3-1a5c-4812-ae27-61bffe5e134c', 'resourceVersion': '175463568', 'creationTimestamp': '2026-06-11T08:38:48Z', 'labels': {'kubernetes.io/metadata.name': 'ovms-smoke', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c34,c24', 'openshift.io/sa.scc.supplemental-groups': '1001170000/10000', 'openshift.io/sa.scc.uid-range': '1001170000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-11T08:38:48Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:38:48Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-11T08:38:48Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-11T08:39:25.850075Z [info ] Wait until Namespace ovms-smoke is deleted [ocp_resources Namespace] 2026-06-11T08:39:25.850226Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:39:33.986981Z [info ] Elapsed time: 8.119933605194092 [0:00:08.119934] [timeout_sampler] _ 15 of 17 completed, 14 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py::TestTensorflowModel::test_tensorflow_inference[tensorflow-raw-rest-deployment]  ------------------------------ test_tensorflow_inference[tensorflow-raw-rest-deployment] ------------------------------ -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:39:33.989814Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:39:33.989956Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:39:33.990077Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:39:33.990204Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:39:33.990371Z [info ] [cached since 646.5s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py::test_tensorflow_inference[tensorflow-raw-rest-deployment]', 1781167173) [sqlalchemy.engine.Engine] 2026-06-11T08:39:33.990787Z [info ] [cached since 646.5s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py::test_tensorflow_inference[tensorflow-raw-rest-deployment]', 1781167173) [sqlalchemy.engine.Engine] 2026-06-11T08:39:33.990923Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:39:33.991043Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:39:33.996163Z [info ] Executing session fixture: root_dir [conftest] 2026-06-11T08:39:33.996455Z [info ] Executing session fixture: triton_runtime_image [conftest] 2026-06-11T08:39:33.996565Z [info ] Executing session fixture: supported_accelerator_type [conftest] 2026-06-11T08:39:33.996668Z [info ] Executing session fixture: s3_models_storage_uri [conftest] 2026-06-11T08:39:33.996770Z [info ] Executing class fixture: triton_rest_serving_runtime_template [conftest] 2026-06-11T08:39:33.996872Z [info ] Create Template triton-rest-runtime-template [ocp_resources Template] 2026-06-11T08:39:33.996983Z [info ] Posting {'apiVersion': 'template.openshift.io/v1', 'kind': 'Template', 'metadata': {'name': 'triton-rest-runtime-template', 'namespace': 'redhat-ods-applications'}, 'objects': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'name': 'triton-rest-runtime', 'annotations': {'prometheus.kserve.io/path': '/metrics', 'prometheus.kserve.io/port': '8002'}}, 'spec': {'containers': [{'name': 'kserve-container', 'image': 'nvcr.io/nvidia/tritonserver:25.02-py3', 'args': ['tritonserver', '--model-store=/mnt/models', '--http-port=8080', '--allow-http=True'], 'ports': [{'name': 'http1', 'containerPort': 8080, 'protocol': 'TCP'}], 'volumeMounts': [], 'resources': {'requests': {'cpu': '1', 'memory': '2Gi'}, 'limits': {'cpu': '1', 'memory': '2Gi'}}}], 'volumes': [], 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'name': 'tensorrt', 'version': '8', 'autoSelect': True, 'priority': 1}, {'name': 'tensorflow', 'version': '1', 'autoSelect': True, 'priority': 1}, {'name': 'tensorflow', 'version': '2', 'autoSelect': True, 'priority': 1}, {'name': 'onnx', 'version': '1', 'autoSelect': True, 'priority': 1}, {'name': 'pytorch', 'version': '1', 'autoSelect': True}, {'name': 'triton', 'version': '2', 'autoSelect': True, 'priority': 1}, {'name': 'xgboost', 'version': '1', 'autoSelect': True}, {'name': 'python', 'version': '1', 'autoSelect': True}]}}], 'parameters': []} [ocp_resources Template] 2026-06-11T08:39:34.049749Z [info ] Executing class fixture: triton_grpc_serving_runtime_template [conftest] 2026-06-11T08:39:34.050043Z [info ] Create Template triton-grpc-runtime-template [ocp_resources Template] 2026-06-11T08:39:34.050320Z [info ] Posting {'apiVersion': 'template.openshift.io/v1', 'kind': 'Template', 'metadata': {'name': 'triton-grpc-runtime-template', 'namespace': 'redhat-ods-applications'}, 'objects': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'name': 'triton-grpc-runtime', 'annotations': {'prometheus.kserve.io/path': '/metrics', 'prometheus.kserve.io/port': '8002'}}, 'spec': {'containers': [{'name': 'kserve-container', 'image': 'nvcr.io/nvidia/tritonserver:25.02-py3', 'args': ['tritonserver', '--model-store=/mnt/models', '--grpc-port=9000', '--allow-grpc=True'], 'ports': [{'name': 'h2c', 'containerPort': 9000, 'protocol': 'TCP'}], 'volumeMounts': [{'name': 'shm', 'mountPath': '/dev/shm'}], 'resources': {'requests': {'cpu': '1', 'memory': '2Gi'}, 'limits': {'cpu': '1', 'memory': '2Gi'}}}], 'volumes': [{'name': 'shm', 'emptyDir': {'medium': 'Memory', 'sizeLimit': '2Gi'}}], 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'name': 'tensorrt', 'version': '8', 'autoSelect': True, 'priority': 1}, {'name': 'tensorflow', 'version': '1', 'autoSelect': True, 'priority': 1}, {'name': 'tensorflow', 'version': '2', 'autoSelect': True, 'priority': 1}, {'name': 'onnx', 'version': '1', 'autoSelect': True, 'priority': 1}, {'name': 'pytorch', 'version': '1', 'autoSelect': True}, {'name': 'triton', 'version': '2', 'autoSelect': True, 'priority': 1}, {'name': 'xgboost', 'version': '1', 'autoSelect': True}, {'name': 'python', 'version': '1', 'autoSelect': True}]}}], 'parameters': []} [ocp_resources Template] 2026-06-11T08:39:34.098577Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-11T08:39:34.098950Z [info ] Create Namespace tensorflow-raw [ocp_resources Namespace] 2026-06-11T08:39:34.099236Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'tensorflow-raw'}, 'spec': {}} [ocp_resources Namespace] 2026-06-11T08:39:34.131799Z [info ] Wait for Namespace tensorflow-raw status to be Active [ocp_resources Namespace] 2026-06-11T08:39:34.132082Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-11T08:39:34.159600Z [info ] Status of Namespace tensorflow-raw is Active [ocp_resources Namespace] 2026-06-11T08:39:34.159886Z [info ] Elapsed time: 5.173683166503906e-05 [0:00:00.000052] [timeout_sampler] 2026-06-11T08:39:34.160131Z [info ] Executing class fixture: protocol [conftest] 2026-06-11T08:39:34.160335Z [info ] Executing class fixture: triton_serving_runtime [conftest] 2026-06-11T08:39:34.193967Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 --- [DuplicateFilter: Last log `kind: Template api version: template.openshift.io/v1` repeated 3 times] [ocp_resources.resource] 2026-06-11T08:39:34.194320Z [info ] Create ServingRuntime triton-rest-runtime [ocp_resources ServingRuntime] 2026-06-11T08:39:34.194473Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'name': 'triton-rest-runtime', 'annotations': {'prometheus.kserve.io/path': '/metrics', 'prometheus.kserve.io/port': '8002'}, 'namespace': 'tensorflow-raw'}, 'spec': {'containers': [{'name': 'kserve-container', 'image': 'nvcr.io/nvidia/tritonserver:25.02-py3', 'args': ['tritonserver', '--model-store=/mnt/models', '--http-port=8080', '--allow-http=True'], 'ports': [{'name': 'http1', 'containerPort': 8080, 'protocol': 'TCP'}], 'volumeMounts': [], 'resources': {'requests': {'cpu': '1', 'memory': '2Gi'}, 'limits': {'cpu': '1', 'memory': '2Gi'}}}], 'volumes': [], 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'name': 'tensorrt', 'version': '8', 'autoSelect': True, 'priority': 1}, {'name': 'tensorflow', 'version': '1', 'autoSelect': True, 'priority': 1}, {'name': 'tensorflow', 'version': '2', 'autoSelect': True, 'priority': 1}, {'name': 'onnx', 'version': '1', 'autoSelect': True, 'priority': 1}, {'name': 'pytorch', 'version': '1', 'autoSelect': True}, {'name': 'triton', 'version': '2', 'autoSelect': True, 'priority': 1}, {'name': 'xgboost', 'version': '1', 'autoSelect': True}, {'name': 'python', 'version': '1', 'autoSelect': True}]}} [ocp_resources ServingRuntime] 2026-06-11T08:39:34.233616Z [info ] Executing class fixture: kserve_s3_secret [conftest] 2026-06-11T08:39:34.233982Z [info ] Create Secret models-bucket-secret [ocp_resources Secret] 2026-06-11T08:39:34.234643Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'models-bucket-secret', 'annotations': {'serving.kserve.io/s3-endpoint': 's3.us-east-2.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-2', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-verifyssl': '0', 'serving.kserve.io/s3-usehttps': '1'}, 'namespace': 'tensorflow-raw'}, 'stringData': '*******'} [ocp_resources Secret] 2026-06-11T08:39:34.269299Z [info ] Wait until Secret models-bucket-secret is created [ocp_resources Secret] 2026-06-11T08:39:34.269472Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-11T08:39:34.283243Z [info ] Elapsed time: 3.8623809814453125e-05 [0:00:00.000039] [timeout_sampler] 2026-06-11T08:39:34.283512Z [info ] Executing class fixture: triton_model_service_account [conftest] 2026-06-11T08:39:34.283646Z [info ] Create ServiceAccount triton-models-bucket-sa [ocp_resources ServiceAccount] 2026-06-11T08:39:34.283753Z [info ] Posting {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'triton-models-bucket-sa', 'namespace': 'tensorflow-raw'}, 'secrets': [{'name': 'models-bucket-secret'}]} [ocp_resources ServiceAccount] 2026-06-11T08:39:34.360294Z [info ] Executing class fixture: triton_inference_service [conftest] 2026-06-11T08:39:34.378706Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:39:34.379109Z [info ] Create InferenceService tensorflow-raw-rest [ocp_resources InferenceService] 2026-06-11T08:39:34.379409Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'tensorflow-raw-rest', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'tensorflow-raw'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'tensorrt'}, 'version': None, 'runtime': 'triton-rest-runtime', 'storageUri': 's3://ods-ci-wisdom/triton/model_repository/', 'resources': {'requests': {'cpu': '1', 'memory': '2Gi'}, 'limits': {'cpu': '2', 'memory': '4Gi'}}}, 'minReplicas': 1, 'serviceAccountName': 'triton-models-bucket-sa'}}} [ocp_resources InferenceService] 2026-06-11T08:39:34.493057Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'triton-rest-runtime'}) [timeout_sampler] 2026-06-11T08:39:34.492272Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:39:35.523513Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:39:35.569941Z [info ] Elapsed time: 1.0313496589660645 [0:00:01.031350] [timeout_sampler] 2026-06-11T08:39:35.582890Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-06-11T08:39:35.584236Z [info ] Waiting for 599.9999976158142 seconds [0:09:59.999998], retry every 10 seconds. (Function: utilities.infra.get_pods_by_isvc_label Kwargs: {'client': , 'isvc': , 'runtime_name': 'triton-rest-runtime'}) [timeout_sampler] 2026-06-11T08:42:57.730666Z [info ] Elapsed time: 202.0747230052948 [0:03:22.074723] [timeout_sampler] 2026-06-11T08:42:57.747486Z [info ] Waiting for 396.74533009529114 seconds [0:06:36.745330], retry every 5 seconds. (Function: ocp_resources.resource.get Kwargs: {'label_selector': 'serving.kserve.io/inferenceservice=tensorflow-raw-rest', 'client': , 'namespace': 'tensorflow-raw'}) [timeout_sampler] 2026-06-11T08:42:57.761916Z [info ] Elapsed time: 9.107589721679688e-05 [0:00:00.000091] [timeout_sampler] 2026-06-11T08:42:57.761140Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-06-11T08:42:57.806260Z [info ] Waiting for 396.68662548065186 seconds [0:06:36.686625], retry every 5 seconds. (Function: utilities.infra..lambda: deployment.instance) [timeout_sampler] 2026-06-11T08:42:57.824750Z [info ] Elapsed time: 6.842613220214844e-05 [0:00:00.000068] [timeout_sampler] 2026-06-11T08:42:57.825128Z [info ] Wait for Deployment tensorflow-raw-rest-predictor to be deployed: True [ocp_resources Deployment] 2026-06-11T08:42:57.825448Z [info ] Waiting for 396.668071269989 seconds [0:06:36.668071], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-06-11T08:42:57.843740Z [info ] Elapsed time: 3.5762786865234375e-05 [0:00:00.000036] [timeout_sampler] 2026-06-11T08:42:57.844128Z [info ] Waiting for 396.64912724494934 seconds [0:06:36.649127], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-06-11T08:42:57.866557Z [info ] Elapsed time: 4.124641418457031e-05 [0:00:00.000041] [timeout_sampler] 2026-06-11T08:42:57.866731Z [info ] Wait for InferenceService/tensorflow-raw-rest's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-11T08:42:57.866842Z [info ] Wait until InferenceService tensorflow-raw-rest is created [ocp_resources InferenceService] 2026-06-11T08:42:57.866946Z [info ] Waiting for 396.6262583732605 seconds [0:06:36.626258], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-11T08:42:57.897720Z [info ] Elapsed time: 2.9087066650390625e-05 [0:00:00.000029] [timeout_sampler] 2026-06-11T08:42:57.897915Z [info ] Waiting for 396.59503626823425 seconds [0:06:36.595036], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-11T08:42:57.915916Z [info ] Elapsed time: 3.933906555175781e-05 [0:00:00.000039] [timeout_sampler] 2026-06-11T08:42:57.944300Z [info ] Waiting for 396.54830408096313 seconds [0:06:36.548304], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-11T08:42:57.971035Z [info ] Elapsed time: 6.270408630371094e-05 [0:00:00.000063] [timeout_sampler] 2026-06-11T08:42:57.996874Z [info ] Executing function fixture: triton_pod_resource [conftest] 2026-06-11T08:42:58.091219Z [info ] Executing function fixture: snapshot [conftest] 2026-06-11T08:42:58.092773Z [info ] Executing function fixture: triton_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- FAILED TEST: TestTensorflowModel.test_tensorflow_inference[tensorflow-raw-rest-deployment] STATUS: FAILED 2026-06-11T08:42:58.203116Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 179, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1720, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 166, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py", line 97, in test_tensorflow_inference validate_inference_request( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ pod_name=triton_pod_resource.name, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<5 lines>... root_dir=root_dir, ^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/utils.py", line 148, in validate_inference_request assert response.get("outputs"), "Response missing outputs" ^^^^^^^^^^^^ AttributeError: 'str' object has no attribute 'get' [conftest] 2026-06-11T08:42:58.205621Z [info ] Must-gather collection is enabled for /home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py::test_tensorflow_inference[tensorflow-raw-rest-deployment]. [conftest] 2026-06-11T08:42:58.205991Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:42:58.210347Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:42:58.210680Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-11T08:42:58.210789Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-11T08:42:58.211171Z [info ] [cached since 586s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py::test_tensorflow_inference[tensorflow-raw-rest-deployment]', 1, 0) [sqlalchemy.engine.Engine] 2026-06-11T08:42:58.211304Z [info ] [cached since 586s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py::test_tensorflow_inference[tensorflow-raw-rest-deployment]', 1, 0) [sqlalchemy.engine.Engine] 2026-06-11T08:42:58.211444Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-11T08:42:58.211553Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-11T08:42:58.211666Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-11T08:42:58.277473Z [info ] Checking subscription rhods-operator [utilities.infra] name=utilities.infra 2026-06-11T08:42:58.291240Z [info ] Expected CSV: rhods-operator.3.4.1 [utilities.infra] name=utilities.infra 2026-06-11T08:42:58.722211Z [info ] Running oc adm must-gather --dest-dir=/home/odh/opendatahub-tests/results/must-gather-collected/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model/TestTensorflowModel/test_tensorflow_inference_tensorflow-raw-rest-deployment_/pytest_exception_interact --since=205s --image=registry.redhat.io/rhoai/odh-must-gather-rhel9@sha256:ec1d4bcb2ba86e7b252cbc95d7eb16e4f468520d76d461f8adf4036746051b15 command [pyhelper_utils.shell] _ TestTensorflowModel.test_tensorflow_inference[tensorflow-raw-rest-deployment] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 179, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1720, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 166, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py", line 97, in test_tensorflow_inference validate_inference_request( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ pod_name=triton_pod_resource.name, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<5 lines>... root_dir=root_dir, ^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/utils.py", line 148, in validate_inference_request assert response.get("outputs"), "Response missing outputs" ^^^^^^^^^^^^ AttributeError: 'str' object has no attribute 'get' _ 16 of 17 completed, 14 Pass, 2 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py::TestTensorflowModel::test_tensorflow_inference[tensorflow-raw-grpc-deployment] 2026-06-11T08:48:09.296528Z [info ] /home/odh/opendatahub-tests/results/must-gather-collected/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model/TestTensorflowModel/test_tensorflow_inference_tensorflow-raw-rest-deployment_/pytest_exception_interact/mg-1781167173.zip is collected successfully [utilities.must_gather_collector] name=utilities.must_gather_collector ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ------------------------------ test_tensorflow_inference[tensorflow-raw-grpc-deployment] ------------------------------ -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T08:48:09.303036Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:48:09.305060Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:48:09.305249Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:48:09.305412Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T08:48:09.305753Z [info ] [cached since 1162s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py::test_tensorflow_inference[tensorflow-raw-grpc-deployment]', 1781167689) [sqlalchemy.engine.Engine] 2026-06-11T08:48:09.305897Z [info ] [cached since 1162s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py::test_tensorflow_inference[tensorflow-raw-grpc-deployment]', 1781167689) [sqlalchemy.engine.Engine] 2026-06-11T08:48:09.306011Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:48:09.306129Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T08:48:09.306238Z [info ] Delete InferenceService tensorflow-raw-rest [ocp_resources InferenceService] 2026-06-11T08:48:09.396675Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'tensorrt', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-11T08:39:34Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:minReplicas': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:resources': {'.': {}, 'f:limits': {'.': {}, 'f:cpu': {}, 'f:memory': {}}, 'f:requests': {'.': {}, 'f:cpu': {}, 'f:memory': {}}}, 'f:runtime': {}, 'f:storageUri': {}}, 'f:serviceAccountName': {}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:39:34Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-11T08:39:34Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-11T08:42:49Z'}], 'name': 'tensorflow-raw-rest', 'namespace': 'tensorflow-raw', 'resourceVersion': '175469973', 'uid': '5451178d-e551-46eb-8787-e01ba8be6ce1'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'minReplicas': 1, 'model': {'modelFormat': {'name': 'tensorrt'}, 'name': '', 'resources': {'limits': {'cpu': '2', 'memory': '4Gi'}, 'requests': {'cpu': '1', 'memory': '2Gi'}}, 'runtime': 'triton-rest-runtime', 'storageUri': 's3://ods-ci-wisdom/triton/model_repository/'}, 'serviceAccountName': 'triton-models-bucket-sa'}}, 'status': {'address': {'url': 'http://tensorflow-raw-rest-predictor.tensorflow-raw.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://tensorflow-raw-rest-predictor-tensorflow-raw.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-11T08:39:34Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-11T08:42:49Z', 'message': 'ReplicaSet "tensorflow-raw-rest-predictor-858c5884c9" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-11T08:42:49Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T08:39:34Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'triton-rest-runtime', 'url': 'http://tensorflow-raw-rest-predictor.tensorflow-raw.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-11T08:48:09.415165Z [info ] Wait until InferenceService tensorflow-raw-rest is deleted [ocp_resources InferenceService] 2026-06-11T08:48:09.415471Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:48:10.445965Z [info ] Elapsed time: 1.0141243934631348 [0:00:01.014124] [timeout_sampler] 2026-06-11T08:48:10.446340Z [info ] Delete ServingRuntime triton-rest-runtime [ocp_resources ServingRuntime] 2026-06-11T08:48:10.476922Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'prometheus.kserve.io/path': '/metrics', 'prometheus.kserve.io/port': '8002'}, 'creationTimestamp': '2026-06-11T08:39:34Z', 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:prometheus.kserve.io/path': {}, 'f:prometheus.kserve.io/port': {}}}, 'f:spec': {'.': {}, 'f:containers': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}, 'f:volumes': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-11T08:39:34Z'}], 'name': 'triton-rest-runtime', 'namespace': 'tensorflow-raw', 'resourceVersion': '175464845', 'uid': '63526023-42cd-4fc8-88d2-a34ad6569fd0'}, 'spec': {'containers': [{'args': ['tritonserver', '--model-store=/mnt/models', '--http-port=8080', '--allow-http=True'], 'image': 'nvcr.io/nvidia/tritonserver:25.02-py3', 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'name': 'http1', 'protocol': 'TCP'}], 'resources': {'limits': {'cpu': '1', 'memory': '2Gi'}, 'requests': {'cpu': '1', 'memory': '2Gi'}}, 'volumeMounts': []}], 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'tensorrt', 'priority': 1, 'version': '8'}, {'autoSelect': True, 'name': 'tensorflow', 'priority': 1, 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'priority': 1, 'version': '2'}, {'autoSelect': True, 'name': 'onnx', 'priority': 1, 'version': '1'}, {'autoSelect': True, 'name': 'pytorch', 'version': '1'}, {'autoSelect': True, 'name': 'triton', 'priority': 1, 'version': '2'}, {'autoSelect': True, 'name': 'xgboost', 'version': '1'}, {'autoSelect': True, 'name': 'python', 'version': '1'}], 'volumes': []}} [ocp_resources ServingRuntime] 2026-06-11T08:48:10.494311Z [info ] Wait until ServingRuntime triton-rest-runtime is deleted [ocp_resources ServingRuntime] 2026-06-11T08:48:10.494506Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-11T08:48:10.509110Z [info ] Elapsed time: 3.7670135498046875e-05 [0:00:00.000038] [timeout_sampler] 2026-06-11T08:48:10.509402Z [info ] Executing class fixture: protocol [conftest] 2026-06-11T08:48:10.509512Z [info ] Executing class fixture: triton_serving_runtime [conftest] 2026-06-11T08:48:10.509626Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-11T08:48:10.549194Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-11T08:48:10.549374Z [info ] Create ServingRuntime triton-grpc-runtime [ocp_resources ServingRuntime] 2026-06-11T08:48:10.549509Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'name': 'triton-grpc-runtime', 'annotations': {'prometheus.kserve.io/path': '/metrics', 'prometheus.kserve.io/port': '8002'}, 'namespace': 'tensorflow-raw'}, 'spec': {'containers': [{'name': 'kserve-container', 'image': 'nvcr.io/nvidia/tritonserver:25.02-py3', 'args': ['tritonserver', '--model-store=/mnt/models', '--grpc-port=9000', '--allow-grpc=True'], 'ports': [{'name': 'h2c', 'containerPort': 9000, 'protocol': 'TCP'}], 'volumeMounts': [{'name': 'shm', 'mountPath': '/dev/shm'}], 'resources': {'requests': {'cpu': '1', 'memory': '2Gi'}, 'limits': {'cpu': '1', 'memory': '2Gi'}}}], 'volumes': [{'name': 'shm', 'emptyDir': {'medium': 'Memory', 'sizeLimit': '2Gi'}}], 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'name': 'tensorrt', 'version': '8', 'autoSelect': True, 'priority': 1}, {'name': 'tensorflow', 'version': '1', 'autoSelect': True, 'priority': 1}, {'name': 'tensorflow', 'version': '2', 'autoSelect': True, 'priority': 1}, {'name': 'onnx', 'version': '1', 'autoSelect': True, 'priority': 1}, {'name': 'pytorch', 'version': '1', 'autoSelect': True}, {'name': 'triton', 'version': '2', 'autoSelect': True, 'priority': 1}, {'name': 'xgboost', 'version': '1', 'autoSelect': True}, {'name': 'python', 'version': '1', 'autoSelect': True}]}} [ocp_resources ServingRuntime] 2026-06-11T08:48:10.585736Z [info ] Executing class fixture: triton_inference_service [conftest] 2026-06-11T08:48:10.600205Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-11T08:48:10.600374Z [info ] Create InferenceService tensorflow-raw-grpc [ocp_resources InferenceService] 2026-06-11T08:48:10.600511Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'tensorflow-raw-grpc', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'tensorflow-raw'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'tensorrt'}, 'version': None, 'runtime': 'triton-grpc-runtime', 'storageUri': 's3://ods-ci-wisdom/triton/model_repository/', 'resources': {'requests': {'cpu': '1', 'memory': '2Gi'}, 'limits': {'cpu': '2', 'memory': '4Gi'}}}, 'minReplicas': 1, 'serviceAccountName': 'triton-models-bucket-sa'}}} [ocp_resources InferenceService] 2026-06-11T08:48:10.713403Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'triton-grpc-runtime'}) [timeout_sampler] 2026-06-11T08:48:10.712628Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:48:11.739635Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-11T08:48:11.793731Z [info ] Elapsed time: 1.0270295143127441 [0:00:01.027030] [timeout_sampler] 2026-06-11T08:48:11.823408Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-06-11T08:48:11.824619Z [info ] Waiting for 599.9999957084656 seconds [0:09:59.999996], retry every 10 seconds. (Function: utilities.infra.get_pods_by_isvc_label Kwargs: {'client': , 'isvc': , 'runtime_name': 'triton-grpc-runtime'}) [timeout_sampler] 2026-06-11T08:48:32.065286Z [info ] Elapsed time: 20.168635845184326 [0:00:20.168636] [timeout_sampler] 2026-06-11T08:48:32.079006Z [info ] Waiting for 578.6343801021576 seconds [0:09:38.634380], retry every 5 seconds. (Function: ocp_resources.resource.get Kwargs: {'label_selector': 'serving.kserve.io/inferenceservice=tensorflow-raw-grpc', 'client': , 'namespace': 'tensorflow-raw'}) [timeout_sampler] 2026-06-11T08:48:32.093979Z [info ] Elapsed time: 7.963180541992188e-05 [0:00:00.000080] [timeout_sampler] 2026-06-11T08:48:32.093250Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-06-11T08:48:32.136835Z [info ] Waiting for 578.5764763355255 seconds [0:09:38.576476], retry every 5 seconds. (Function: utilities.infra..lambda: deployment.instance) [timeout_sampler] 2026-06-11T08:48:32.150156Z [info ] Elapsed time: 0.00010728836059570312 [0:00:00.000107] [timeout_sampler] 2026-06-11T08:48:32.150317Z [info ] Wait for Deployment tensorflow-raw-grpc-predictor to be deployed: True [ocp_resources Deployment] 2026-06-11T08:48:32.150554Z [info ] Waiting for 578.5628688335419 seconds [0:09:38.562869], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-06-11T08:48:32.163897Z [info ] Elapsed time: 3.0517578125e-05 [0:00:00.000031] [timeout_sampler] 2026-06-11T08:48:32.164038Z [info ] Waiting for 578.5491180419922 seconds [0:09:38.549118], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-06-11T08:48:32.177863Z [info ] Elapsed time: 3.0040740966796875e-05 [0:00:00.000030] [timeout_sampler] 2026-06-11T08:48:32.178020Z [info ] Wait for InferenceService/tensorflow-raw-grpc's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-11T08:48:32.178205Z [info ] Wait until InferenceService tensorflow-raw-grpc is created [ocp_resources InferenceService] 2026-06-11T08:48:32.178320Z [info ] Waiting for 578.5351889133453 seconds [0:09:38.535189], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-11T08:48:32.192626Z [info ] Elapsed time: 2.86102294921875e-05 [0:00:00.000029] [timeout_sampler] 2026-06-11T08:48:32.192923Z [info ] Waiting for 578.5209150314331 seconds [0:09:38.520915], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-11T08:48:32.206702Z [info ] Elapsed time: 6.008148193359375e-05 [0:00:00.000060] [timeout_sampler] 2026-06-11T08:48:32.220876Z [info ] Waiting for 578.4923369884491 seconds [0:09:38.492337], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-11T08:48:32.233760Z [info ] Elapsed time: 5.793571472167969e-05 [0:00:00.000058] [timeout_sampler] 2026-06-11T08:48:32.248016Z [info ] Executing function fixture: triton_pod_resource [conftest] 2026-06-11T08:48:32.291092Z [info ] Executing function fixture: snapshot [conftest] 2026-06-11T08:48:32.291266Z [info ] Executing function fixture: triton_response_snapshot [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- FAILED TEST: TestTensorflowModel.test_tensorflow_inference[tensorflow-raw-grpc-deployment] STATUS: FAILED 2026-06-11T08:48:32.316520Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 179, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1720, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 166, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py", line 97, in test_tensorflow_inference validate_inference_request( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ pod_name=triton_pod_resource.name, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<5 lines>... root_dir=root_dir, ^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/utils.py", line 148, in validate_inference_request assert response.get("outputs"), "Response missing outputs" ^^^^^^^^^^^^ AttributeError: 'str' object has no attribute 'get' [conftest] 2026-06-11T08:48:32.321122Z [info ] Must-gather collection is enabled for /home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py::test_tensorflow_inference[tensorflow-raw-grpc-deployment]. [conftest] 2026-06-11T08:48:32.321488Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:48:32.321745Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T08:48:32.322297Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-11T08:48:32.322429Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-11T08:48:32.322567Z [info ] [cached since 920.1s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py::test_tensorflow_inference[tensorflow-raw-grpc-deployment]', 1, 0) [sqlalchemy.engine.Engine] 2026-06-11T08:48:32.322692Z [info ] [cached since 920.1s ago] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model.py::test_tensorflow_inference[tensorflow-raw-grpc-deployment]', 1, 0) [sqlalchemy.engine.Engine] 2026-06-11T08:48:32.322804Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-11T08:48:32.322908Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-11T08:48:32.323017Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-11T08:48:32.394319Z [info ] Checking subscription rhods-operator [utilities.infra] name=utilities.infra 2026-06-11T08:48:32.408243Z [info ] Expected CSV: rhods-operator.3.4.1 [utilities.infra] name=utilities.infra 2026-06-11T08:48:32.574730Z [info ] Running oc adm must-gather --dest-dir=/home/odh/opendatahub-tests/results/must-gather-collected/model_serving/model_runtime/triton/basic_model_deployment/test_tensorflow_model/TestTensorflowModel/test_tensorflow_inference_tensorflow-raw-grpc-deployment_/pytest_exception_interact --since=300s --image=registry.redhat.io/rhoai/odh-must-gather-rhel9@sha256:ec1d4bcb2ba86e7b252cbc95d7eb16e4f468520d76d461f8adf4036746051b15 command [pyhelper_utils.shell]