--- BVT pytest attempt 2026-06-14T08:34:53.566031+00:00 (/.local/bin/uv run pytest...) --- 2026-06-14T08:34:54.784590Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-06-14T08:34:54.794380Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.794806Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.795708Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.795921Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.796095Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.796617Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.796920Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.799192Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.799391Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.799569Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.799723Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.799881Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.800056Z [info ] [no key 0.00005s] ()  [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.800204Z [info ] [no key 0.00005s] ()  [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.800334Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.800502Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-14T08:34:54.800623Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-14T08:34:54.807613Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-06-14T08:34:54.922155Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-14T08:34:54.959484Z [info ] Running model registry tests against default namespace [conftest] 2026-06-14T08:34:54.959791Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] ============================= test session starts ============================== platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.14.5', 'Platform': 'Linux-5.14.0-427.115.1.el9_4.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.0.3', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.13.0', 'asyncio': '1.3.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.3.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.1.0'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.13.0, asyncio-1.3.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.3.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.1.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... collected 49 items / 48 deselected / 1 selected tests/model_serving/model_runtime/mlserver/s3/test_mlserver_s3.py::TestMLServerModels::test_mlserver_model_inference[sklearn-s3-RawDeployment]  ------------------------------- test_mlserver_model_inference[sklearn-s3-RawDeployment] ------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-14T08:34:55.182956Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() started [sqlalchemy.orm.mapper.Mapper] 2026-06-14T08:34:55.183956Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop id [sqlalchemy.orm.mapper.Mapper] 2026-06-14T08:34:55.184361Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop test_name [sqlalchemy.orm.mapper.Mapper] 2026-06-14T08:34:55.184544Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop start_time [sqlalchemy.orm.mapper.Mapper] 2026-06-14T08:34:55.185345Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() complete [sqlalchemy.orm.mapper.Mapper] 2026-06-14T08:34:55.185478Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-14T08:34:55.185596Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-14T08:34:55.185713Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-14T08:34:55.185852Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-14T08:34:55.186009Z [info ] [generated in 0.00012s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/s3/test_mlserver_s3.py::test_mlserver_model_inference[sklearn-s3-RawDeployment]', 1781426095) [sqlalchemy.engine.Engine] 2026-06-14T08:34:55.186149Z [info ] [generated in 0.00012s] ('/home/odh/opendatahub-tests/tests/model_serving/model_runtime/mlserver/s3/test_mlserver_s3.py::test_mlserver_model_inference[sklearn-s3-RawDeployment]', 1781426095) [sqlalchemy.engine.Engine] 2026-06-14T08:34:55.186274Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-14T08:34:55.186393Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-14T08:34:55.187857Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-06-14T08:34:55.187991Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-06-14T08:34:55.188122Z [info ] Executing session fixture: admin_client [conftest] 2026-06-14T08:34:55.188243Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-14T08:34:55.196488Z [info ] Executing session fixture: dsc_resource [conftest] 2026-06-14T08:34:55.196614Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-06-14T08:34:55.259591Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-06-14T08:34:55.259916Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-06-14T08:34:55.260832Z [info ] Executing session fixture: os_path_environment [conftest] 2026-06-14T08:34:55.261058Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-06-14T08:34:55.261226Z [info ] Executing session fixture: bin_directory [conftest] 2026-06-14T08:34:55.262339Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-06-14T08:34:55.262706Z [info ] kind: ConsoleCLIDownload api version: console.openshift.io/v1 [ocp_resources.resource] 2026-06-14T08:34:55.291500Z [info ] All oc console cli download links: ['https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar'] [utilities.infra] name=utilities.infra 2026-06-14T08:34:55.291621Z [info ] Downloading archive using: url=https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar [utilities.infra] name=utilities.infra 2026-06-14T08:35:13.201203Z [info ] Extract the downloaded archive. [utilities.infra] name=utilities.infra 2026-06-14T08:35:13.333361Z [info ] Downloaded file: ['oc']  [utilities.infra] name=utilities.infra 2026-06-14T08:35:13.364950Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-06-14T08:35:13.360400Z [info ] OC binary path: /tmp/pytest-8QqekZ5yq6UCyHrHcEoddv/bin0/oc [tests.conftest] name=tests.conftest 2026-06-14T08:35:13.360493Z [info ] Adding /tmp/pytest-8QqekZ5yq6UCyHrHcEoddv/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-06-14T08:35:13.365404Z [info ] Executing session fixture: nodes [conftest] 2026-06-14T08:35:13.412224Z [info ] Executing session fixture: dsci_resource [conftest] 2026-06-14T08:35:13.412394Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-14T08:35:13.425951Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-06-14T08:35:13.426116Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-06-14T08:35:13.426329Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-06-14T08:35:13.425238Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-06-14T08:35:13.426553Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-06-14T08:35:13.516161Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-06-14T08:35:13.589450Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsci_status_ready Kwargs: {'dsci_resource': }) [timeout_sampler] 2026-06-14T08:35:13.588862Z [info ] Wait for DSCI default-dsci to be in Ready status. [utilities.infra] name=utilities.infra 2026-06-14T08:35:13.589730Z [info ] Get DSCInitialization default-dsci status [ocp_resources DSCInitialization] 2026-06-14T08:35:13.601236Z [info ] Elapsed time: 0.00014901161193847656 [0:00:00.000149] [timeout_sampler] 2026-06-14T08:35:13.601377Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': }) [timeout_sampler] 2026-06-14T08:35:13.600663Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T08:35:13.601623Z [info ] Get DataScienceCluster default-dsc status [ocp_resources DataScienceCluster] 2026-06-14T08:35:13.617528Z [info ] Elapsed time: 4.00543212890625e-05 [0:00:00.000040] [timeout_sampler] 2026-06-14T08:35:13.617688Z [info ] Executing session fixture: autouse_fixtures [conftest] 2026-06-14T08:35:13.617865Z [info ] Executing session fixture: pytestconfig [conftest] 2026-06-14T08:35:13.618020Z [info ] Executing session fixture: aws_access_key_id [conftest] 2026-06-14T08:35:13.618146Z [info ] Executing session fixture: aws_secret_access_key [conftest] 2026-06-14T08:35:13.618267Z [info ] Executing session fixture: ci_s3_bucket_endpoint [conftest] 2026-06-14T08:35:13.618420Z [info ] Executing session fixture: valid_aws_config [conftest] 2026-06-14T08:35:13.676899Z [info ] AWS credentials validated successfully via STS GetCallerIdentity [tests.conftest] name=tests.conftest 2026-06-14T08:35:13.680626Z [info ] Executing session fixture: teardown_resources [conftest] 2026-06-14T08:35:13.680769Z [info ] Executing session fixture: mlserver_runtime_image [conftest] 2026-06-14T08:35:13.680948Z [info ] Executing session fixture: models_s3_bucket_name [conftest] 2026-06-14T08:35:13.681094Z [info ] Executing session fixture: s3_models_storage_uri [conftest] 2026-06-14T08:35:13.681219Z [info ] Executing session fixture: models_s3_bucket_region [conftest] 2026-06-14T08:35:13.681332Z [info ] Executing session fixture: models_s3_bucket_endpoint [conftest] 2026-06-14T08:35:13.681441Z [info ] Executing class fixture: model_namespace [conftest] 2026-06-14T08:35:13.681776Z [info ] Create Namespace sklearn-s3  [ocp_resources Namespace] 2026-06-14T08:35:13.681911Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'sklearn-s3'}, 'spec': {}} [ocp_resources Namespace] 2026-06-14T08:35:13.710951Z [info ] Wait for Namespace sklearn-s3 status to be Active [ocp_resources Namespace] 2026-06-14T08:35:13.711094Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-06-14T08:35:13.726278Z [info ] Status of Namespace sklearn-s3 is Active [ocp_resources Namespace] 2026-06-14T08:35:13.726419Z [info ] Elapsed time: 4.3392181396484375e-05 [0:00:00.000043] [timeout_sampler] 2026-06-14T08:35:13.726607Z [info ] Executing class fixture: mlserver_serving_runtime [conftest] 2026-06-14T08:35:13.726741Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-06-14T08:35:13.772082Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-06-14T08:35:13.772250Z [info ] Create ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-14T08:35:13.772463Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'mlserver-runtime', 'namespace': 'sklearn-s3'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-14T08:35:13.810087Z [info ] Executing class fixture: kserve_s3_secret [conftest] 2026-06-14T08:35:13.810302Z [info ] Create Secret models-bucket-secret [ocp_resources Secret] 2026-06-14T08:35:13.810429Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'models-bucket-secret', 'annotations': {'serving.kserve.io/s3-endpoint': 's3.us-east-2.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-2', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-verifyssl': '0', 'serving.kserve.io/s3-usehttps': '1'}, 'namespace': 'sklearn-s3'}, 'stringData': '*******'} [ocp_resources Secret] 2026-06-14T08:35:13.838152Z [info ] Wait until Secret models-bucket-secret is created [ocp_resources Secret] 2026-06-14T08:35:13.838300Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-14T08:35:13.849277Z [info ] Elapsed time: 4.458427429199219e-05 [0:00:00.000045] [timeout_sampler] 2026-06-14T08:35:13.849420Z [info ] Executing class fixture: mlserver_model_service_account [conftest] 2026-06-14T08:35:13.849586Z [info ] Create ServiceAccount mlserver-models-bucket-sa [ocp_resources ServiceAccount] 2026-06-14T08:35:13.849700Z [info ] Posting {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'mlserver-models-bucket-sa', 'namespace': 'sklearn-s3'}, 'secrets': [{'name': 'models-bucket-secret'}]} [ocp_resources ServiceAccount] 2026-06-14T08:35:13.877038Z [info ] Executing class fixture: mlserver_inference_service [conftest] 2026-06-14T08:35:13.889658Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-06-14T08:35:13.889887Z [info ] Create InferenceService sklearn [ocp_resources InferenceService] 2026-06-14T08:35:13.890050Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'sklearn', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'sklearn-s3'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'sklearn'}, 'version': None, 'runtime': 'mlserver-runtime', 'storageUri': 's3://ods-ci-wisdom/mlserver/model_repository/sklearn/', 'resources': {'requests': {'cpu': '1', 'memory': '2Gi'}, 'limits': {'cpu': '2', 'memory': '4Gi'}}}, 'minReplicas': 1, 'serviceAccountName': 'mlserver-models-bucket-sa'}}} [ocp_resources InferenceService] 2026-06-14T08:35:13.978953Z [info ] Waiting for 30 seconds [0:00:30], retry every 1 seconds. (Function: utilities.infra.wait_for_isvc_pods Kwargs: {'client': , 'isvc': , 'runtime_name': 'mlserver-runtime'}) [timeout_sampler] 2026-06-14T08:35:13.978474Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-14T08:35:15.001753Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-06-14T08:35:15.037397Z [info ] Elapsed time: 1.0232784748077393 [0:00:01.023278] [timeout_sampler] 2026-06-14T08:35:15.050475Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-06-14T08:35:15.051148Z [info ] Waiting for 899.9999890327454 seconds [0:14:59.999989], retry every 10 seconds. (Function: utilities.infra.get_pods_by_isvc_label Kwargs: {'client': , 'isvc': , 'runtime_name': 'mlserver-runtime'}) [timeout_sampler] 2026-06-14T08:35:25.212763Z [info ] Elapsed time: 10.102772235870361 [0:00:10.102772] [timeout_sampler] 2026-06-14T08:35:25.223915Z [info ] Waiting for 888.7553021907806 seconds [0:14:48.755302], retry every 5 seconds. (Function: ocp_resources.resource.get Kwargs: {'label_selector': 'serving.kserve.io/inferenceservice=sklearn', 'client': , 'namespace': 'sklearn-s3'}) [timeout_sampler] 2026-06-14T08:35:25.224089Z [info ] kind: Deployment api version: apps/v1 [ocp_resources.resource] 2026-06-14T08:35:25.235246Z [info ] Elapsed time: 7.82012939453125e-05 [0:00:00.000078] [timeout_sampler] 2026-06-14T08:35:25.234703Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-06-14T08:35:25.269473Z [info ] Waiting for 888.7093782424927 seconds [0:14:48.709378], retry every 5 seconds. (Function: utilities.infra..lambda: deployment.instance) [timeout_sampler] 2026-06-14T08:35:25.280765Z [info ] Elapsed time: 5.4836273193359375e-05 [0:00:00.000055] [timeout_sampler] 2026-06-14T08:35:25.281003Z [info ] Wait for Deployment sklearn-predictor to be deployed: True [ocp_resources Deployment] 2026-06-14T08:35:25.281165Z [info ] Waiting for 888.6980817317963 seconds [0:14:48.698082], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-06-14T08:35:25.292466Z [info ] Elapsed time: 2.956390380859375e-05 [0:00:00.000030] [timeout_sampler] 2026-06-14T08:35:25.292665Z [info ] Waiting for 888.6863813400269 seconds [0:14:48.686381], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-06-14T08:35:25.304295Z [info ] Elapsed time: 3.528594970703125e-05 [0:00:00.000035] [timeout_sampler] 2026-06-14T08:35:25.304444Z [info ] Wait for InferenceService/sklearn's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-06-14T08:35:25.304578Z [info ] Wait until InferenceService sklearn is created [ocp_resources InferenceService] 2026-06-14T08:35:25.304707Z [info ] Waiting for 888.6746790409088 seconds [0:14:48.674679], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-14T08:35:25.316523Z [info ] Elapsed time: 6.031990051269531e-05 [0:00:00.000060] [timeout_sampler] 2026-06-14T08:35:25.316674Z [info ] Waiting for 888.6623764038086 seconds [0:14:48.662376], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance) [timeout_sampler] 2026-06-14T08:35:25.327786Z [info ] Elapsed time: 3.075599670410156e-05 [0:00:00.000031] [timeout_sampler] 2026-06-14T08:35:25.339372Z [info ] Waiting for 888.6395664215088 seconds [0:14:48.639566], retry every 5 seconds. (Function: utilities.inference_utils._is_model_loaded) [timeout_sampler] 2026-06-14T08:35:25.351314Z [info ] Elapsed time: 7.867813110351562e-05 [0:00:00.000079] [timeout_sampler] 2026-06-14T08:35:25.362363Z [info ] Executing function fixture: mlserver_pod_resource [conftest] 2026-06-14T08:35:25.387167Z [info ] Executing function fixture: snapshot [conftest] 2026-06-14T08:35:25.387940Z [info ] Executing function fixture: mlserver_response_snapshot [conftest] 2026-06-14T08:35:25.389506Z [info ] Executing function fixture: model_format [conftest] --------------------------------------------------------- CALL --------------------------------------------------------- [2026-06-14T08:35:25Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed PASSED TEST: TestMLServerModels.test_mlserver_model_inference[sklearn-s3-RawDeployment] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-06-14T08:35:25.545001Z [info ] Delete InferenceService sklearn [ocp_resources InferenceService] 2026-06-14T08:35:25.572808Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'sklearn', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-06-14T08:35:13Z', 'finalizers': ['odh.inferenceservice.finalizers', 'inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:minReplicas': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:resources': {'.': {}, 'f:limits': {'.': {}, 'f:cpu': {}, 'f:memory': {}}, 'f:requests': {'.': {}, 'f:cpu': {}, 'f:memory': {}}}, 'f:runtime': {}, 'f:storageUri': {}}, 'f:serviceAccountName': {}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-14T08:35:13Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-06-14T08:35:14Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:modelClass': {}}}, 'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-06-14T08:35:24Z'}], 'name': 'sklearn', 'namespace': 'sklearn-s3', 'resourceVersion': '182488920', 'uid': 'eedcdd65-c0a1-4389-aad4-9869e7e813a3'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'minReplicas': 1, 'model': {'modelFormat': {'name': 'sklearn'}, 'name': '', 'resources': {'limits': {'cpu': '2', 'memory': '4Gi'}, 'requests': {'cpu': '1', 'memory': '2Gi'}}, 'runtime': 'mlserver-runtime', 'storageUri': 's3://ods-ci-wisdom/mlserver/model_repository/sklearn/'}, 'serviceAccountName': 'mlserver-models-bucket-sa'}}, 'status': {'address': {'url': 'http://sklearn-predictor.sklearn-s3.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://sklearn-predictor-sklearn-s3.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-06-14T08:35:14Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-06-14T08:35:24Z', 'message': 'ReplicaSet "sklearn-predictor-8594f6766d" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-06-14T08:35:24Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-14T08:35:14Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'mlserver-runtime', 'url': 'http://sklearn-predictor.sklearn-s3.svc.cluster.local'}} [ocp_resources InferenceService] 2026-06-14T08:35:25.590518Z [info ] Wait until InferenceService sklearn is deleted [ocp_resources InferenceService] 2026-06-14T08:35:25.590811Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-14T08:35:26.620858Z [info ] Elapsed time: 1.013197898864746 [0:00:01.013198] [timeout_sampler] 2026-06-14T08:35:26.621057Z [info ] Delete ServingRuntime mlserver-runtime [ocp_resources ServingRuntime] 2026-06-14T08:35:26.667231Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/runtime-version': '1.7.1', 'openshift.io/display-name': 'MLServer ServingRuntime for KServe', 'serving.kserve.io/server-type': 'mlserver'}, 'creationTimestamp': '2026-06-14T08:35:13Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}, 'f:serving.kserve.io/server-type': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:monitoring.opendatahub.io/scrape': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-06-14T08:35:13Z'}], 'name': 'mlserver-runtime', 'namespace': 'sklearn-s3', 'resourceVersion': '182488553', 'uid': 'e35955e2-b7af-44c9-afd3-4d2dc26cb94b'}, 'spec': {'annotations': {'monitoring.opendatahub.io/scrape': 'true', 'opendatahub.io/kserve-runtime': 'mlserver', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8082'}, 'containers': [{'env': [{'name': 'MLSERVER_MODEL_IMPLEMENTATION', 'value': '{{.Labels.modelClass}}'}, {'name': 'MLSERVER_HTTP_PORT', 'value': '8080'}, {'name': 'MLSERVER_MODELS_DIR', 'value': '/mnt/models'}], 'image': 'registry.redhat.io/rhoai/odh-mlserver-rhel9@sha256:d76bea18afe7b361847babb7a8ebc51fdbcd8164435f1bcb971e1701ba1bc595', 'livenessProbe': {'failureThreshold': 6, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 20, 'periodSeconds': 10, 'timeoutSeconds': 5}, 'name': 'kserve-container', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'readinessProbe': {'failureThreshold': 3, 'httpGet': {'path': '/v2/models/{{.Name}}/ready', 'port': 8080}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 5}, 'securityContext': {'allowPrivilegeEscalation': False, 'capabilities': {'drop': ['ALL']}, 'privileged': False, 'runAsNonRoot': True}, 'startupProbe': {'exec': {'command': ['/bin/sh', '-c', '[ -n "$(ls -A /mnt/models 2>/dev/null)" ]\n']}, 'failureThreshold': 1, 'initialDelaySeconds': 1, 'periodSeconds': 1}}], 'multiModel': False, 'protocolVersions': ['v2'], 'supportedModelFormats': [{'name': 'sklearn', 'version': '0'}, {'name': 'sklearn', 'version': '1'}, {'name': 'xgboost', 'version': '1'}, {'name': 'xgboost', 'version': '2'}, {'name': 'lightgbm', 'version': '3'}, {'name': 'lightgbm', 'version': '4'}, {'name': 'onnx', 'version': '1'}]}} [ocp_resources ServingRuntime] 2026-06-14T08:35:26.681243Z [info ] Wait until ServingRuntime mlserver-runtime is deleted [ocp_resources ServingRuntime] 2026-06-14T08:35:26.681399Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-14T08:35:26.694932Z [info ] Elapsed time: 8.058547973632812e-05 [0:00:00.000081] [timeout_sampler] 2026-06-14T08:35:26.695100Z [info ] Delete ServiceAccount mlserver-models-bucket-sa [ocp_resources ServiceAccount] 2026-06-14T08:35:26.720732Z [info ] Deleting {'kind': 'ServiceAccount', 'apiVersion': 'v1', 'metadata': {'name': 'mlserver-models-bucket-sa', 'namespace': 'sklearn-s3', 'uid': 'b0ce6d29-1430-400c-8016-a3f24ecb659b', 'resourceVersion': '182488557', 'creationTimestamp': '2026-06-14T08:35:13Z', 'annotations': {'openshift.io/internal-registry-pull-secret-ref': 'mlserver-models-bucket-sa-dockercfg-jm88g'}, 'managedFields': [{'manager': 'openshift.io/image-registry-pull-secrets_service-account-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-14T08:35:13Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:imagePullSecrets': {}, 'f:metadata': {'f:annotations': {'f:openshift.io/internal-registry-pull-secret-ref': {}}}, 'f:secrets': {'k:{"name":"mlserver-models-bucket-sa-dockercfg-jm88g"}': {}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-14T08:35:13Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:secrets': {'.': {}, 'k:{"name":"models-bucket-secret"}': {}}}}]}, 'secrets': [{'name': 'models-bucket-secret'}, {'name': 'mlserver-models-bucket-sa-dockercfg-jm88g'}], 'imagePullSecrets': [{'name': 'mlserver-models-bucket-sa-dockercfg-jm88g'}]} [ocp_resources ServiceAccount] 2026-06-14T08:35:26.736238Z [info ] Wait until ServiceAccount mlserver-models-bucket-sa is deleted [ocp_resources ServiceAccount] 2026-06-14T08:35:26.736411Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-14T08:35:26.748381Z [info ] Elapsed time: 8.392333984375e-05 [0:00:00.000084] [timeout_sampler] 2026-06-14T08:35:26.748552Z [info ] Delete Secret models-bucket-secret [ocp_resources Secret] 2026-06-14T08:35:26.782899Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'models-bucket-secret', 'namespace': 'sklearn-s3', 'uid': '13fabbd9-e3c6-490e-a9d6-fb0258e55b68', 'resourceVersion': '182488554', 'creationTimestamp': '2026-06-14T08:35:13Z', 'annotations': {'serving.kserve.io/s3-endpoint': 's3.us-east-2.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-2', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-usehttps': '1', 'serving.kserve.io/s3-verifyssl': '0'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-14T08:35:13Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:AWS_ACCESS_KEY_ID': {}, 'f:AWS_SECRET_ACCESS_KEY': {}}, 'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/s3-endpoint': {}, 'f:serving.kserve.io/s3-region': {}, 'f:serving.kserve.io/s3-useanoncredential': {}, 'f:serving.kserve.io/s3-usehttps': {}, 'f:serving.kserve.io/s3-verifyssl': {}}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-06-14T08:35:26.798411Z [info ] Wait until Secret models-bucket-secret is deleted [ocp_resources Secret] 2026-06-14T08:35:26.798616Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-14T08:35:26.810362Z [info ] Elapsed time: 3.218650817871094e-05 [0:00:00.000032] [timeout_sampler] 2026-06-14T08:35:26.856907Z [info ] Delete Namespace sklearn-s3  [ocp_resources Namespace] 2026-06-14T08:35:26.884369Z [info ] Deleting {'kind': 'Namespace', 'apiVersion': 'v1', 'metadata': {'name': 'sklearn-s3', 'uid': '624d0668-8101-43be-b3d9-0584bf8f1ff7', 'resourceVersion': '182488538', 'creationTimestamp': '2026-06-14T08:35:13Z', 'labels': {'kubernetes.io/metadata.name': 'sklearn-s3', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/sa.scc.mcs': 's0:c31,c25', 'openshift.io/sa.scc.supplemental-groups': '1000980000/10000', 'openshift.io/sa.scc.uid-range': '1000980000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-06-14T08:35:13Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-14T08:35:13Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-06-14T08:35:13Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Namespace] 2026-06-14T08:35:26.899387Z [info ] Wait until Namespace sklearn-s3 is deleted [ocp_resources Namespace] 2026-06-14T08:35:26.899551Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-06-14T08:35:35.015392Z [info ] Elapsed time: 8.103002548217773 [0:00:08.103003] [timeout_sampler] _ 1 of 1 completed, 1 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ ================= 1 passed, 48 deselected, 1 warning in 40.03s ================= =============================== warnings summary =============================== tests/model_serving/model_runtime/mlserver/s3/test_mlserver_s3.py::TestMLServerModels::test_mlserver_model_inference[sklearn-s3-RawDeployment] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ------------ generated xml file: /artifacts/model_runtime-smoke.xml ------------ ================= 1 passed, 48 deselected, 1 warning in 40.04s =================