--- BVT pytest attempt 2026-07-09T16:02:36.935973+00:00 (/.local/bin/uv --directory /home/odh/opendatahub-tests...) --- 2026-07-09T16:02:38.970677Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-07-09T16:02:38.971300Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-07.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-07.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-07.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-07.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-07.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-07-09T16:02:39.091336Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-07.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-07.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-07.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-07.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-07-09T16:02:39.143152Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-07.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( ============================= test session starts ============================== platform linux -- Python 3.14.6, pytest-9.1.1, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python3 cachedir: .pytest_cache metadata: {'Python': '3.14.6', 'Platform': 'Linux-5.14.0-570.120.1.el9_6.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.1.1', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.14.0', 'langsmith': '0.9.0', 'asyncio': '1.4.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.5.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.3.2'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.14.0, langsmith-0.9.0, asyncio-1.4.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.5.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.3.2 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... collected 224 items / 219 deselected / 5 selected tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_model_authentication_using_rest_raw[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0]  ----- test_model_authentication_using_rest_raw[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] ----- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-07-09T16:02:39.612118Z [info ] Executing session fixture: admin_client [conftest] 2026-07-09T16:02:39.613368Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:02:39.613521Z [info ] Executing session fixture: dsc_resource [conftest] 2026-07-09T16:02:39.613661Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-07-09T16:02:39.668766Z [info ] Executing package fixture: enabled_kserve_in_dsc [conftest] 2026-07-09T16:02:39.681695Z [warning ] Component kserve was already set to managementState Managed [utilities.data_science_cluster_utils] name=utilities.data_science_cluster_utils 2026-07-09T16:02:39.682689Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-07-09T16:02:39.682939Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-07-09T16:02:39.683158Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-07-09T16:02:39.683306Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-07-09T16:02:39.684124Z [info ] Executing session fixture: os_path_environment [conftest] 2026-07-09T16:02:39.684240Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-07-09T16:02:39.684371Z [info ] Executing session fixture: bin_directory [conftest] 2026-07-09T16:02:39.685246Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-07-09T16:02:39.705717Z [info ] kind: ConsoleCLIDownload api version: console.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:02:39.728419Z [info ] All oc console cli download links: ['https://downloads-openshift-console.apps.ods-qe-psi-07.osp.rh-ods.com/amd64/linux/oc.tar'] [utilities.infra] name=utilities.infra 2026-07-09T16:02:39.728536Z [info ] Downloading archive using: url=https://downloads-openshift-console.apps.ods-qe-psi-07.osp.rh-ods.com/amd64/linux/oc.tar [utilities.infra] name=utilities.infra 2026-07-09T16:03:00.278185Z [info ] Extract the downloaded archive. [utilities.infra] name=utilities.infra 2026-07-09T16:03:00.420099Z [info ] Downloaded file: ['oc']  [utilities.infra] name=utilities.infra 2026-07-09T16:03:01.236070Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-07-09T16:03:01.233747Z [info ] OC binary path: /tmp/pytest-YrHoZPqsvNfbJKwjcJf4uA/bin0/oc [tests.conftest] name=tests.conftest 2026-07-09T16:03:01.233862Z [info ] Adding /tmp/pytest-YrHoZPqsvNfbJKwjcJf4uA/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-07-09T16:03:01.236544Z [info ] Executing session fixture: nodes [conftest] 2026-07-09T16:03:02.246424Z [info ] Executing session fixture: dsci_resource [conftest] 2026-07-09T16:03:02.246601Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-07-09T16:03:02.259842Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-07-09T16:03:02.260190Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-07-09T16:03:02.260396Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-07-09T16:03:02.258768Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-07-09T16:03:02.260623Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-07-09T16:03:02.359442Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-07-09T16:03:02.467228Z [warning ] Skipping RHOAI resource checks, got --cluster-sanity-skip-rhoai-check [utilities.infra] name=utilities.infra 2026-07-09T16:03:02.468832Z [info ] Executing session fixture: autouse_fixtures [conftest] 2026-07-09T16:03:02.468987Z [info ] Executing session fixture: kserve_health_check [conftest] 2026-07-09T16:03:02.499837Z [info ] kind: Deployment api version: apps/v1 [ocp_resources.resource] 2026-07-09T16:03:02.549842Z [info ] KServe component health check passed [tests.model_serving.model_server.kserve.conftest] 2026-07-09T16:03:02.550013Z [info ] Executing session fixture: pytestconfig [conftest] 2026-07-09T16:03:02.550148Z [info ] Executing session fixture: aws_access_key_id [conftest] 2026-07-09T16:03:02.550276Z [info ] Executing session fixture: aws_secret_access_key [conftest] 2026-07-09T16:03:02.550381Z [info ] Executing session fixture: ci_s3_bucket_endpoint [conftest] 2026-07-09T16:03:02.550483Z [info ] Executing session fixture: valid_aws_config [conftest] 2026-07-09T16:03:02.606508Z [info ] AWS credentials validated successfully via STS GetCallerIdentity [tests.conftest] name=tests.conftest 2026-07-09T16:03:02.608548Z [info ] Executing session fixture: use_unprivileged_client [conftest] 2026-07-09T16:03:02.608663Z [info ] Executing session fixture: kubconfig_filepath [conftest] 2026-07-09T16:03:02.608768Z [info ] Executing session fixture: cluster_authentication [conftest] 2026-07-09T16:03:02.608928Z [info ] kind: Authentication api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:02.627717Z [info ] Executing session fixture: is_byoidc [conftest] 2026-07-09T16:03:02.645047Z [info ] Executing session fixture: non_admin_user_password [conftest] 2026-07-09T16:03:02.645217Z [info ] Executing session fixture: unprivileged_client [conftest] 2026-07-09T16:03:02.642568Z [warning ] Unprivileged client is not enabled, using admin client [tests.conftest] name=tests.conftest 2026-07-09T16:03:02.645543Z [info ] Executing session fixture: teardown_resources [conftest] 2026-07-09T16:03:02.645695Z [info ] Executing session fixture: ci_s3_bucket_name [conftest] 2026-07-09T16:03:02.645883Z [info ] Executing session fixture: ci_s3_bucket_region [conftest] 2026-07-09T16:03:02.646037Z [info ] Executing session fixture: models_s3_bucket_name [conftest] 2026-07-09T16:03:02.646232Z [info ] Executing session fixture: models_s3_bucket_region [conftest] 2026-07-09T16:03:02.646382Z [info ] Executing session fixture: models_s3_bucket_endpoint [conftest] 2026-07-09T16:03:02.646595Z [info ] Executing class fixture: unprivileged_model_namespace [conftest] 2026-07-09T16:03:02.646706Z [info ] kind: ProjectRequest api version: project.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:02.646840Z [info ] Create ProjectRequest test-kserve-raw-token-authentication [ocp_resources ProjectRequest] 2026-07-09T16:03:02.646971Z [info ] Posting {'apiVersion': 'project.openshift.io/v1', 'kind': 'ProjectRequest', 'metadata': {'name': 'test-kserve-raw-token-authentication'}} [ocp_resources ProjectRequest] 2026-07-09T16:03:02.779577Z [info ] kind: Project api version: project.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:02.779730Z [info ] Wait for Project test-kserve-raw-token-authentication status to be Active [ocp_resources Project] 2026-07-09T16:03:02.797750Z [info ] Status of Project test-kserve-raw-token-authentication is Active [ocp_resources Project] 2026-07-09T16:03:02.797925Z [info ] Executing class fixture: http_s3_ovms_serving_runtime [conftest] 2026-07-09T16:03:02.798042Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:02.906613Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:03:02.906796Z [info ] Create ServingRuntime http-mnist-runtime [ocp_resources ServingRuntime] 2026-07-09T16:03:02.907076Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'http-mnist-runtime', 'namespace': 'test-kserve-raw-token-authentication'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:5bb57ef927e7481cfc768993faa8e1e70b27154ecc81a88991bd5428001c1ea6', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-07-09T16:03:02.937688Z [info ] Executing class fixture: ci_endpoint_s3_secret [conftest] 2026-07-09T16:03:02.958592Z [info ] Create Secret ci-bucket-secret [ocp_resources Secret] 2026-07-09T16:03:02.958763Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'ci-bucket-secret', 'labels': {'opendatahub.io/managed': 'true', 'opendatahub.io/dashboard': 'true'}, 'annotations': {'opendatahub.io/connection-type': 's3', 'serving.kserve.io/s3-endpoint': 's3.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-1', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-verifyssl': '0', 'serving.kserve.io/s3-usehttps': '1'}, 'namespace': 'test-kserve-raw-token-authentication'}, 'data': '*******'} [ocp_resources Secret] 2026-07-09T16:03:02.995452Z [info ] Wait until Secret ci-bucket-secret is created [ocp_resources Secret] 2026-07-09T16:03:03.010507Z [info ] Executing class fixture: models_endpoint_s3_secret [conftest] 2026-07-09T16:03:03.028823Z [info ] Create Secret models-bucket-secret [ocp_resources Secret] 2026-07-09T16:03:03.029039Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'models-bucket-secret', 'labels': {'opendatahub.io/managed': 'true', 'opendatahub.io/dashboard': 'true'}, 'annotations': {'opendatahub.io/connection-type': 's3', 'serving.kserve.io/s3-endpoint': 's3.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-1', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-verifyssl': '0', 'serving.kserve.io/s3-usehttps': '1'}, 'namespace': 'test-kserve-raw-token-authentication'}, 'data': '*******'} [ocp_resources Secret] 2026-07-09T16:03:03.070353Z [info ] Wait until Secret models-bucket-secret is created [ocp_resources Secret] 2026-07-09T16:03:03.088902Z [info ] Executing class fixture: model_service_account [conftest] 2026-07-09T16:03:03.089068Z [info ] Create ServiceAccount models-bucket-sa [ocp_resources ServiceAccount] 2026-07-09T16:03:03.089207Z [info ] Posting {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'models-bucket-sa', 'namespace': 'test-kserve-raw-token-authentication'}, 'secrets': [{'name': 'models-bucket-secret'}]} [ocp_resources ServiceAccount] 2026-07-09T16:03:03.117198Z [info ] Executing class fixture: http_s3_ovms_raw_inference_service [conftest] 2026-07-09T16:03:03.130792Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-07-09T16:03:03.131011Z [info ] Create InferenceService http-onnx [ocp_resources InferenceService] 2026-07-09T16:03:03.131251Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'http-onnx', 'labels': {'networking.kserve.io/visibility': 'exposed'}, 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment', 'security.opendatahub.io/enable-auth': 'true'}, 'namespace': 'test-kserve-raw-token-authentication'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'openvino_ir'}, 'version': None, 'runtime': 'http-mnist-runtime', 'storage': {'key': 'ci-bucket-secret', 'path': '/test-dir/'}}, 'serviceAccountName': 'models-bucket-sa'}}} [ocp_resources InferenceService] 2026-07-09T16:03:03.200204Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-07-09T16:03:04.230902Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-07-09T16:03:04.287120Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-07-09T16:03:24.615917Z [info ] kind: Deployment api version: apps/v1 [ocp_resources.resource] 2026-07-09T16:03:24.672503Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-07-09T16:03:24.720031Z [info ] Wait for Deployment http-onnx-predictor to be deployed: True [ocp_resources Deployment] 2026-07-09T16:03:24.747228Z [info ] Wait for InferenceService/http-onnx's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-07-09T16:03:24.747395Z [info ] Wait until InferenceService http-onnx is created [ocp_resources InferenceService] 2026-07-09T16:03:24.803829Z [info ] Executing class fixture: http_raw_view_role [conftest] 2026-07-09T16:03:24.804042Z [info ] kind: Role api version: rbac.authorization.k8s.io/v1 [ocp_resources.resource] 2026-07-09T16:03:24.804176Z [info ] Create Role http-onnx-view  [ocp_resources Role] 2026-07-09T16:03:24.804309Z [info ] Posting {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'Role', 'metadata': {'name': 'http-onnx-view', 'namespace': 'test-kserve-raw-token-authentication'}, 'rules': [{'apiGroups': ['serving.kserve.io'], 'resources': ['inferenceservices'], 'verbs': ['get'], 'resourceNames': ['http-onnx']}]} [ocp_resources Role] 2026-07-09T16:03:24.830188Z [info ] Executing class fixture: http_raw_role_binding [conftest] 2026-07-09T16:03:24.830346Z [info ] kind: RoleBinding api version: rbac.authorization.k8s.io/v1 [ocp_resources.resource] 2026-07-09T16:03:24.830491Z [info ] Create RoleBinding http-models-bucket-sa-view [ocp_resources RoleBinding] 2026-07-09T16:03:24.830633Z [info ] Posting {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'RoleBinding', 'metadata': {'name': 'http-models-bucket-sa-view', 'namespace': 'test-kserve-raw-token-authentication'}, 'subjects': [{'kind': 'ServiceAccount', 'name': 'models-bucket-sa'}], 'roleRef': {'kind': 'Role', 'name': 'http-onnx-view', 'apiGroup': 'rbac.authorization.k8s.io'}} [ocp_resources RoleBinding] 2026-07-09T16:03:24.857502Z [info ] Executing class fixture: http_raw_inference_token [conftest] 2026-07-09T16:03:24.857666Z [info ] Running oc create token -n test-kserve-raw-token-authentication models-bucket-sa command [pyhelper_utils.shell] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-07-09T16:03:25.019429Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:03:25.058661Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:03:25.063034Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:25.161790Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:25.174634Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:03:25.190546Z [info ] Running Hide By User command  [pyhelper_utils.shell] 2026-07-09T16:03:25.283923Z [info ] Inference output: HTTP/1.1 404 Not Found access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * content-length: 53 content-type: application/json; charset=utf-8 date: Thu, 09 Jul 2026 16:03:25 GMT set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=6b188d6608209d3fd187e37f91a72f96; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils FAILED _ TestKserveTokenAuthenticationRawForRest.test_model_authentication_using_rest_raw[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py", line 39, in test_model_authentication_using_rest_raw verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=http_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... token=http_raw_inference_token, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} _ 1 of 5 completed, 0 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_disabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0]  TEST: TestKserveTokenAuthenticationRawForRest.test_model_authentication_using_rest_raw[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] STATUS: FAILED 2026-07-09T16:03:25.298493Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py", line 39, in test_model_authentication_using_rest_raw verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=http_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... token=http_raw_inference_token, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} [conftest] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ------ test_disabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] ------ -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-07-09T16:03:25.299187Z [info ] Executing function fixture: patched_remove_raw_authentication_isvc [conftest] 2026-07-09T16:03:25.299299Z [info ] ResourceEdit: Backing up old data [ocp_resources.resource] 2026-07-09T16:03:25.309954Z [info ] ResourceEdits: Updating data for resource InferenceService http-onnx [ocp_resources.resource] 2026-07-09T16:03:25.310123Z [info ] Update InferenceService http-onnx: {'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, 'name': 'http-onnx'}} [ocp_resources InferenceService] 2026-07-09T16:03:25.337272Z [info ] Wait for InferenceService/http-onnx's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-07-09T16:03:25.337441Z [info ] Wait until InferenceService http-onnx is created [ocp_resources InferenceService] 2026-07-09T16:03:25.389900Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-07-09T16:03:26.534193Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:03:26.571626Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:03:26.575902Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:26.638953Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:26.649667Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:03:26.739484Z [info ] Inference output: HTTP/1.0 400 Bad Request set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=6b188d6608209d3fd187e37f91a72f96; path=/; HttpOnly; Secure; SameSite=None Client sent an HTTP request to an HTTPS server.  [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:03:31.771447Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:03:31.814937Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:03:31.821148Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:31.879956Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:31.892231Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:03:31.959536Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=2ee8361a66aa9f1608c97866250875c1; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:03:36.985462Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:03:37.022376Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:03:37.031290Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:37.104631Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:37.339004Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:03:42.409170Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None connection: close {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:03:47.433853Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:03:47.473683Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:03:47.479976Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:47.550482Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:47.562213Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:03:47.626446Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:03:52.654968Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:03:52.691867Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:03:52.698124Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:52.773489Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:52.789567Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:03:52.857461Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:03:57.883100Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:03:57.919246Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:03:57.926560Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:57.995793Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:03:58.007780Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:03:58.066712Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:04:03.101046Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:04:03.167075Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:04:03.173477Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:03.244203Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:03.264003Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:04:03.338059Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:04:08.363354Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:04:08.397161Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:04:08.404162Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:08.462184Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:08.473364Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:04:08.550401Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:04:13.576417Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:04:13.609707Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:04:13.615718Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:13.679527Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:13.695289Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:04:13.760516Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:04:18.787100Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:04:18.826530Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:04:18.832604Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:18.898294Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:18.912307Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:04:18.980796Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:04:24.008578Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:04:24.054086Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:04:24.060549Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:24.140773Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:24.156478Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:04:24.225772Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:04:29.251544Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:04:29.286159Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:04:29.295378Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:29.355071Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:29.366784Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:04:29.430231Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:04:34.456278Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:04:34.488607Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:04:34.494870Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:34.559671Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:34.572073Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:04:34.631638Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:04:39.657378Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:04:39.692928Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:04:39.697637Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:39.772142Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:39.784855Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:04:39.844785Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:04:44.869046Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:04:44.904381Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:04:44.908571Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:44.980425Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:44.996257Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:04:45.058418Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:04:50.082756Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:04:50.121946Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:04:50.128420Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:50.187277Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:50.198701Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:04:50.256581Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:04:55.287669Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:04:55.329114Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:04:55.335271Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:55.398546Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:04:55.409891Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:04:55.475775Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:05:00.501078Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:05:00.534620Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:05:00.540665Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:00.597091Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:00.608513Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:05:00.664103Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:05:05.689202Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:05:05.727607Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:05:05.732329Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:05.792198Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:05.804099Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:05:05.863906Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:05:10.889500Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:05:10.923027Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:05:10.929544Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:11.121985Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:11.134119Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:05:11.198606Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:05:16.231101Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:05:16.273508Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:05:16.278078Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:16.345850Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:16.363821Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:05:16.441776Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:05:21.466742Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:05:21.501409Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:05:21.507552Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:21.573346Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:21.586101Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:05:21.663094Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:05:26.689941Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:05:26.733248Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:05:26.739413Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:26.803511Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:26.817461Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:05:26.890684Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:05:31.915161Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:05:31.949066Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:05:31.955143Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:32.015067Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:32.027842Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:05:32.102000Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:05:37.146155Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:05:37.188360Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:05:37.195285Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:37.488705Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:37.500095Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:05:37.567627Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:05:42.594786Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:05:42.629477Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:05:42.635708Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:42.700051Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:42.714976Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:05:42.783507Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:05:47.820002Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:05:47.874897Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:05:47.879214Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:47.948610Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:47.965936Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:05:48.041374Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:05:53.066268Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:05:53.099667Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:05:53.109005Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:53.168528Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:53.181256Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:05:53.255350Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:05:58.285842Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:05:58.319614Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:05:58.326439Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:58.414235Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:05:58.425583Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:05:58.499831Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:06:03.528246Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:06:03.569984Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:06:03.576055Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:03.652222Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:03.664417Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:06:03.727384Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:06:08.752717Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:06:08.785631Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:06:08.791849Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:08.857084Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:08.869278Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:06:08.923261Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:06:13.949152Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:06:13.982548Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:06:13.988932Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:14.048153Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:14.060210Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:06:14.118241Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:06:19.144630Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:06:19.181916Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:06:19.188628Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:19.263764Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:19.281644Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:06:19.344335Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:06:24.374110Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:06:24.415073Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:06:24.419468Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:24.485863Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:24.497396Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:06:24.563134Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:06:29.587190Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:06:29.620058Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:06:29.626673Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:29.690919Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:29.702750Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:06:29.764102Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:06:34.788967Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:06:34.821926Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:06:34.828088Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:34.890372Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:34.902682Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:06:34.963498Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:06:39.989840Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:06:40.027515Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:06:40.033896Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:40.100211Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:40.114363Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:06:40.180022Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:06:45.214894Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:06:45.266156Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:06:45.272358Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:45.354413Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:45.367553Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:06:45.435657Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:06:50.467940Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:06:50.502885Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:06:50.509314Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:50.576140Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:50.593689Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:06:50.667222Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:06:55.693609Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:06:55.727853Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:06:55.734614Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:55.797355Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:06:55.808775Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:06:55.966294Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:07:00.991375Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:07:01.029541Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:07:01.036016Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:01.100485Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:01.112661Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:07:01.176369Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:07:06.202903Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:07:06.236309Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:07:06.243118Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:06.310008Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:06.322087Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:07:06.389640Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:07:11.414783Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:07:11.449995Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:07:11.454413Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:11.521772Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:11.534549Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:07:11.604784Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:07:16.631512Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:07:16.665873Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:07:16.672428Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:16.959434Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:16.980594Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:07:17.041358Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:07:22.068084Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:07:22.101930Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:07:22.108224Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:22.174086Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:22.190422Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:07:22.490397Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:07:27.529793Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:07:27.584851Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:07:27.590853Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:27.684215Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:27.701283Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:07:27.782211Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:07:32.807256Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:07:32.840372Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:07:32.846670Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:32.925080Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:32.944170Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:07:33.021387Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:07:38.060883Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:07:38.094993Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:07:38.101408Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:38.174706Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:38.187710Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:07:38.260876Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:07:43.290693Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:07:43.324888Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:07:43.331126Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:43.398220Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:43.410118Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:07:43.480847Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:07:48.506819Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:07:48.540603Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:07:48.546631Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:48.611992Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:48.623778Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:07:48.689895Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:07:53.715162Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:07:53.759534Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:07:53.763832Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:53.847567Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:53.865669Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:07:53.924488Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:07:58.963253Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:07:59.007674Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:07:59.014873Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:59.092876Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:07:59.106773Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:07:59.189693Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:08:04.215120Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:08:04.255655Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:08:04.261607Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:04.347700Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:04.365386Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:08:04.430715Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:08:09.462511Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:08:09.510626Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:08:09.516767Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:09.600224Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:09.618867Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:08:09.685107Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:08:14.709587Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:08:14.744793Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:08:14.750965Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:14.812758Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:14.825376Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:08:14.889309Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:08:19.920696Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:08:19.955194Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:08:19.961131Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:20.019262Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:20.030369Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:08:21.099630Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:08:26.131586Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:08:26.183438Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:08:26.189885Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:26.267657Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:26.292062Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:08:26.364340Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=0398d2c82455165ed3e8414074914300; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:08:31.364972Z [error ] Timeout: InferenceService http-onnx in test-kserve-raw-token-authentication external HTTPS infer not ready within 300s (token=no) [tests.model_serving.model_server.utils] name=tests.model_serving.model_server.utils 2026-07-09T16:08:31.366701Z [info ] ResourceEdits: Restoring data for resource InferenceService http-onnx [ocp_resources.resource] 2026-07-09T16:08:31.366928Z [info ] Update InferenceService http-onnx: {'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'true'}, 'name': 'http-onnx'}} [ocp_resources InferenceService] ERROR _ ERROR at setup of TestKserveTokenAuthenticationRawForRest.test_disabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 170, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 536, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1710, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 806, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 630, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1232, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 924, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1328, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 998, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/conftest.py", line 100, in patched_remove_raw_authentication_isvc wait_for_raw_isvc_https_infer_ready(isvc=http_s3_ovms_raw_inference_service, token=None) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 234, in wait_for_raw_isvc_https_infer_ready for ok in TimeoutSampler(wait_timeout=timeout, sleep=sleep, func=_https_infer_ok): ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 300 Function: tests.model_serving.model_server.utils._https_infer_ok Last exception: N/A. _ 2 of 5 completed, 0 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 1 Error, 0 ReRun _ tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_re_enabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0]  TEST: TestKserveTokenAuthenticationRawForRest.test_disabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] [setup] STATUS: ERROR 2026-07-09T16:08:31.420115Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 170, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 536, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1710, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 806, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 630, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1232, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 924, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1328, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 998, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/conftest.py", line 100, in patched_remove_raw_authentication_isvc wait_for_raw_isvc_https_infer_ready(isvc=http_s3_ovms_raw_inference_service, token=None) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 234, in wait_for_raw_isvc_https_infer_ready for ok in TimeoutSampler(wait_timeout=timeout, sleep=sleep, func=_https_infer_ok): ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 300 Function: tests.model_serving.model_server.utils._https_infer_ok Last exception: N/A. [conftest] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ----- test_re_enabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] ----- -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-07-09T16:08:31.446117Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:08:31.487476Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:08:31.493729Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:31.550530Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:31.564453Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:08:36.654481Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:08:36.658761Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:36.725446Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:36.737235Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:08:36.827410Z [info ] Inference output: HTTP/1.1 404 Not Found access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * content-length: 53 content-type: application/json; charset=utf-8 date: Thu, 09 Jul 2026 16:08:36 GMT set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=b532b526069210ebd8a0cf71427ca555; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils FAILED _ TestKserveTokenAuthenticationRawForRest.test_re_enabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py", line 61, in test_re_enabled_raw_model_authentication verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=http_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... token=http_raw_inference_token, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} _ 3 of 5 completed, 0 Pass, 2 Fail, 0 Skip, 0 XPass, 0 XFail, 1 Error, 0 ReRun _ tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_cross_model_authentication_raw[http_s3_ovms_raw_inference_service_20-unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0]  TEST: TestKserveTokenAuthenticationRawForRest.test_re_enabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] STATUS: FAILED 2026-07-09T16:08:36.834309Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py", line 61, in test_re_enabled_raw_model_authentication verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=http_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... token=http_raw_inference_token, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} [conftest] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  test_cross_model_authentication_raw[http_s3_ovms_raw_inference_service_20-unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0]  -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-07-09T16:08:36.834912Z [info ] Executing class fixture: model_service_account_2 [conftest] 2026-07-09T16:08:36.835048Z [info ] Create ServiceAccount models-bucket-sa-2 [ocp_resources ServiceAccount] 2026-07-09T16:08:36.835165Z [info ] Posting {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'models-bucket-sa-2', 'namespace': 'test-kserve-raw-token-authentication'}, 'secrets': [{'name': 'models-bucket-secret'}]} [ocp_resources ServiceAccount] 2026-07-09T16:08:36.863046Z [info ] Executing class fixture: http_s3_ovms_raw_inference_service_2 [conftest] 2026-07-09T16:08:36.877074Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-07-09T16:08:36.877207Z [info ] Create InferenceService http-onnx-2 [ocp_resources InferenceService] 2026-07-09T16:08:36.877336Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'http-onnx-2', 'labels': {'networking.kserve.io/visibility': 'exposed'}, 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment', 'security.opendatahub.io/enable-auth': 'true'}, 'namespace': 'test-kserve-raw-token-authentication'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'openvino_ir'}, 'version': None, 'runtime': 'http-mnist-runtime', 'storage': {'key': 'ci-bucket-secret', 'path': '/test-dir/'}}, 'serviceAccountName': 'models-bucket-sa-2'}}} [ocp_resources InferenceService] 2026-07-09T16:08:36.921554Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-07-09T16:08:37.943158Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-07-09T16:08:37.998543Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-07-09T16:08:48.185156Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-07-09T16:08:48.231348Z [info ] Wait for Deployment http-onnx-2-predictor to be deployed: True [ocp_resources Deployment] 2026-07-09T16:08:48.254759Z [info ] Wait for InferenceService/http-onnx-2's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-07-09T16:08:48.255039Z [info ] Wait until InferenceService http-onnx-2 is created [ocp_resources InferenceService] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-07-09T16:08:48.337127Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:08:48.374526Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:08:48.380592Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:48.453846Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:48.465653Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:08:54.591286Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:08:54.597504Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:54.666083Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:08:54.724852Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils PASSED2026-07-09T16:08:54.811076Z [info ] Inference output: HTTP/1.1 403 Forbidden content-type: text/plain; charset=utf-8 x-content-type-options: nosniff date: Thu, 09 Jul 2026 16:08:54 GMT content-length: 145 set-cookie: 69f764f782e5acf3035b89de67f2b2b6=89c1e043e25cf3f9258c9bc581ad11d1; path=/; HttpOnly; Secure; SameSite=None Forbidden (user=system:serviceaccount:test-kserve-raw-token-authentication:models-bucket-sa, verb=get, resource=inferenceservices, subresource=)  [utilities.inference_utils] name=utilities.inference_utils  TEST: TestKserveTokenAuthenticationRawForRest.test_cross_model_authentication_raw[http_s3_ovms_raw_inference_service_20-unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-07-09T16:08:54.813401Z [info ] Delete InferenceService http-onnx-2 [ocp_resources InferenceService] 2026-07-09T16:08:54.860836Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'openvino_ir', 'security.opendatahub.io/enable-auth': 'true', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-07-09T16:08:36Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'labels': {'networking.kserve.io/visibility': 'exposed'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:security.opendatahub.io/enable-auth': {}, 'f:serving.kserve.io/deploymentMode': {}}, 'f:labels': {'.': {}, 'f:networking.kserve.io/visibility': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storage': {'.': {}, 'f:key': {}, 'f:path': {}}}, 'f:serviceAccountName': {}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-07-09T16:08:36Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-07-09T16:08:37Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-07-09T16:08:47Z'}], 'name': 'http-onnx-2', 'namespace': 'test-kserve-raw-token-authentication', 'resourceVersion': '385935103', 'uid': '855bbb4d-631e-4705-9e49-1f26ac2dd90a'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'openvino_ir'}, 'name': '', 'resources': {}, 'runtime': 'http-mnist-runtime', 'storage': {'key': 'ci-bucket-secret', 'path': '/test-dir/'}}, 'serviceAccountName': 'models-bucket-sa-2'}}, 'status': {'address': {'url': 'https://http-onnx-2-predictor.test-kserve-raw-token-authentication.svc.cluster.local:8443'}, 'components': {'predictor': {'url': 'http://http-onnx-2-predictor-test-kserve-raw-token-authentication.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-07-09T16:08:37Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-07-09T16:08:47Z', 'message': 'ReplicaSet "http-onnx-2-predictor-59996c845b" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-07-09T16:08:47Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T16:08:37Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'http-mnist-runtime', 'url': 'https://http-onnx-2-test-kserve-raw-token-authentication.apps.ods-qe-psi-07.osp.rh-ods.com'}} [ocp_resources InferenceService] 2026-07-09T16:08:54.887373Z [info ] Wait until InferenceService http-onnx-2 is deleted [ocp_resources InferenceService] 2026-07-09T16:08:55.927194Z [info ] Delete ServiceAccount models-bucket-sa-2 [ocp_resources ServiceAccount] 2026-07-09T16:08:55.957191Z [info ] Deleting {'kind': 'ServiceAccount', 'apiVersion': 'v1', 'metadata': {'name': 'models-bucket-sa-2', 'namespace': 'test-kserve-raw-token-authentication', 'uid': 'b1a33031-039d-48d2-8750-8f3193ee1688', 'resourceVersion': '385934819', 'creationTimestamp': '2026-07-09T16:08:36Z', 'annotations': {'openshift.io/internal-registry-pull-secret-ref': 'models-bucket-sa-2-dockercfg-s6d6k'}, 'managedFields': [{'manager': 'openshift.io/image-registry-pull-secrets_service-account-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-07-09T16:08:36Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:imagePullSecrets': {}, 'f:metadata': {'f:annotations': {'f:openshift.io/internal-registry-pull-secret-ref': {}}}, 'f:secrets': {'k:{"name":"models-bucket-sa-2-dockercfg-s6d6k"}': {}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:08:36Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:secrets': {'.': {}, 'k:{"name":"models-bucket-secret"}': {}}}}]}, 'secrets': [{'name': 'models-bucket-secret'}, {'name': 'models-bucket-sa-2-dockercfg-s6d6k'}], 'imagePullSecrets': [{'name': 'models-bucket-sa-2-dockercfg-s6d6k'}]} [ocp_resources ServiceAccount] 2026-07-09T16:08:55.971118Z [info ] Wait until ServiceAccount models-bucket-sa-2 is deleted [ocp_resources ServiceAccount] 2026-07-09T16:08:55.983600Z [info ] Delete RoleBinding http-models-bucket-sa-view [ocp_resources RoleBinding] 2026-07-09T16:08:56.006092Z [info ] Deleting {'kind': 'RoleBinding', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'metadata': {'name': 'http-models-bucket-sa-view', 'namespace': 'test-kserve-raw-token-authentication', 'uid': '532e9bf0-fdab-4c6f-b2d4-727cd6a287d3', 'resourceVersion': '385929642', 'creationTimestamp': '2026-07-09T16:03:24Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'time': '2026-07-09T16:03:24Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:roleRef': {}, 'f:subjects': {}}}]}, 'subjects': [{'kind': 'ServiceAccount', 'name': 'models-bucket-sa'}], 'roleRef': {'apiGroup': 'rbac.authorization.k8s.io', 'kind': 'Role', 'name': 'http-onnx-view'}} [ocp_resources RoleBinding] 2026-07-09T16:08:56.021008Z [info ] Wait until RoleBinding http-models-bucket-sa-view is deleted [ocp_resources RoleBinding] 2026-07-09T16:08:56.033246Z [info ] Delete Role http-onnx-view  [ocp_resources Role] 2026-07-09T16:08:56.057290Z [info ] Deleting {'kind': 'Role', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'metadata': {'name': 'http-onnx-view', 'namespace': 'test-kserve-raw-token-authentication', 'uid': 'ec4ddc2e-e8d3-4df7-a78f-c9b67676a92d', 'resourceVersion': '385929640', 'creationTimestamp': '2026-07-09T16:03:24Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'time': '2026-07-09T16:03:24Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:rules': {}}}]}, 'rules': [{'verbs': ['get'], 'apiGroups': ['serving.kserve.io'], 'resources': ['inferenceservices'], 'resourceNames': ['http-onnx']}]} [ocp_resources Role] 2026-07-09T16:08:56.073193Z [info ] Wait until Role http-onnx-view is deleted [ocp_resources Role] 2026-07-09T16:08:56.084613Z [info ] Delete InferenceService http-onnx [ocp_resources InferenceService] 2026-07-09T16:08:56.110455Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'openvino_ir', 'security.opendatahub.io/enable-auth': 'true', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-07-09T16:03:03Z', 'finalizers': ['odh.inferenceservice.finalizers', 'inferenceservice.finalizers'], 'generation': 1, 'labels': {'networking.kserve.io/visibility': 'exposed'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-07-09T16:03:03Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:security.opendatahub.io/enable-auth': {}, 'f:serving.kserve.io/deploymentMode': {}}, 'f:labels': {'.': {}, 'f:networking.kserve.io/visibility': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storage': {'.': {}, 'f:key': {}, 'f:path': {}}}, 'f:serviceAccountName': {}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-07-09T16:08:31Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-07-09T16:08:42Z'}], 'name': 'http-onnx', 'namespace': 'test-kserve-raw-token-authentication', 'resourceVersion': '385934995', 'uid': '20e89e98-5498-43f4-b751-c9e8d99dc218'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'openvino_ir'}, 'name': '', 'resources': {}, 'runtime': 'http-mnist-runtime', 'storage': {'key': 'ci-bucket-secret', 'path': '/test-dir/'}}, 'serviceAccountName': 'models-bucket-sa'}}, 'status': {'address': {'url': 'https://http-onnx-predictor.test-kserve-raw-token-authentication.svc.cluster.local:8443'}, 'components': {'predictor': {'url': 'http://http-onnx-predictor-test-kserve-raw-token-authentication.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-07-09T16:03:03Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-07-09T16:08:42Z', 'message': 'ReplicaSet "http-onnx-predictor-66d9b4d854" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-07-09T16:08:42Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T16:03:03Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 2}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 3, 'servingRuntimeName': 'http-mnist-runtime', 'url': 'https://http-onnx-test-kserve-raw-token-authentication.apps.ods-qe-psi-07.osp.rh-ods.com'}} [ocp_resources InferenceService] 2026-07-09T16:08:56.126697Z [info ] Wait until InferenceService http-onnx is deleted [ocp_resources InferenceService] 2026-07-09T16:08:57.152139Z [info ] Delete ServiceAccount models-bucket-sa [ocp_resources ServiceAccount] 2026-07-09T16:08:57.187396Z [info ] Deleting {'kind': 'ServiceAccount', 'apiVersion': 'v1', 'metadata': {'name': 'models-bucket-sa', 'namespace': 'test-kserve-raw-token-authentication', 'uid': '98b8c42b-b7d7-425b-9164-114138fa044a', 'resourceVersion': '385929198', 'creationTimestamp': '2026-07-09T16:03:03Z', 'annotations': {'openshift.io/internal-registry-pull-secret-ref': 'models-bucket-sa-dockercfg-shgt4'}, 'managedFields': [{'manager': 'openshift.io/image-registry-pull-secrets_service-account-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-07-09T16:03:03Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:imagePullSecrets': {}, 'f:metadata': {'f:annotations': {'f:openshift.io/internal-registry-pull-secret-ref': {}}}, 'f:secrets': {'k:{"name":"models-bucket-sa-dockercfg-shgt4"}': {}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:03:03Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:secrets': {'.': {}, 'k:{"name":"models-bucket-secret"}': {}}}}]}, 'secrets': [{'name': 'models-bucket-secret'}, {'name': 'models-bucket-sa-dockercfg-shgt4'}], 'imagePullSecrets': [{'name': 'models-bucket-sa-dockercfg-shgt4'}]} [ocp_resources ServiceAccount] 2026-07-09T16:08:57.203040Z [info ] Wait until ServiceAccount models-bucket-sa is deleted [ocp_resources ServiceAccount] 2026-07-09T16:08:57.216465Z [info ] Delete Secret models-bucket-secret [ocp_resources Secret] 2026-07-09T16:08:57.248136Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'models-bucket-secret', 'namespace': 'test-kserve-raw-token-authentication', 'uid': '4c910d89-a440-4165-a273-ee433580b7ba', 'resourceVersion': '385929193', 'creationTimestamp': '2026-07-09T16:03:03Z', 'labels': {'opendatahub.io/dashboard': 'true', 'opendatahub.io/managed': 'true'}, 'annotations': {'opendatahub.io/connection-type': 's3', 'serving.kserve.io/s3-endpoint': 's3.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-1', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-usehttps': '1', 'serving.kserve.io/s3-verifyssl': '0'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:03:03Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:AWS_ACCESS_KEY_ID': {}, 'f:AWS_DEFAULT_REGION': {}, 'f:AWS_S3_BUCKET': {}, 'f:AWS_S3_ENDPOINT': {}, 'f:AWS_SECRET_ACCESS_KEY': {}}, 'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/connection-type': {}, 'f:serving.kserve.io/s3-endpoint': {}, 'f:serving.kserve.io/s3-region': {}, 'f:serving.kserve.io/s3-useanoncredential': {}, 'f:serving.kserve.io/s3-usehttps': {}, 'f:serving.kserve.io/s3-verifyssl': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}, 'f:opendatahub.io/managed': {}}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-09T16:08:57.264372Z [info ] Wait until Secret models-bucket-secret is deleted [ocp_resources Secret] 2026-07-09T16:08:57.276993Z [info ] Delete Secret ci-bucket-secret [ocp_resources Secret] 2026-07-09T16:08:57.307145Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'ci-bucket-secret', 'namespace': 'test-kserve-raw-token-authentication', 'uid': '27915976-e52a-4cac-ab1e-951145cd36b7', 'resourceVersion': '385929189', 'creationTimestamp': '2026-07-09T16:03:02Z', 'labels': {'opendatahub.io/dashboard': 'true', 'opendatahub.io/managed': 'true'}, 'annotations': {'opendatahub.io/connection-type': 's3', 'serving.kserve.io/s3-endpoint': 's3.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-1', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-usehttps': '1', 'serving.kserve.io/s3-verifyssl': '0'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:03:02Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:AWS_ACCESS_KEY_ID': {}, 'f:AWS_DEFAULT_REGION': {}, 'f:AWS_S3_BUCKET': {}, 'f:AWS_S3_ENDPOINT': {}, 'f:AWS_SECRET_ACCESS_KEY': {}}, 'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/connection-type': {}, 'f:serving.kserve.io/s3-endpoint': {}, 'f:serving.kserve.io/s3-region': {}, 'f:serving.kserve.io/s3-useanoncredential': {}, 'f:serving.kserve.io/s3-usehttps': {}, 'f:serving.kserve.io/s3-verifyssl': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}, 'f:opendatahub.io/managed': {}}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-09T16:08:57.320305Z [info ] Wait until Secret ci-bucket-secret is deleted [ocp_resources Secret] 2026-07-09T16:08:57.331982Z [info ] Delete ServingRuntime http-mnist-runtime [ocp_resources ServingRuntime] 2026-07-09T16:08:57.358126Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'creationTimestamp': '2026-07-09T16:03:02Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/recommended-accelerators': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-07-09T16:03:02Z'}], 'name': 'http-mnist-runtime', 'namespace': 'test-kserve-raw-token-authentication', 'resourceVersion': '385929187', 'uid': 'e22d6151-6336-4ba0-8f33-660e8085d2a6'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:5bb57ef927e7481cfc768993faa8e1e70b27154ecc81a88991bd5428001c1ea6', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-07-09T16:08:57.373127Z [info ] Wait until ServingRuntime http-mnist-runtime is deleted [ocp_resources ServingRuntime] 2026-07-09T16:08:57.409928Z [info ] Delete Project test-kserve-raw-token-authentication [ocp_resources Project] 2026-07-09T16:08:57.444629Z [info ] Deleting {'kind': 'Project', 'apiVersion': 'project.openshift.io/v1', 'metadata': {'name': 'test-kserve-raw-token-authentication', 'uid': '0371d3f7-f926-4e6e-a876-c713c5844c55', 'resourceVersion': '385929170', 'creationTimestamp': '2026-07-09T16:03:02Z', 'labels': {'kubernetes.io/metadata.name': 'test-kserve-raw-token-authentication', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/description': '', 'openshift.io/display-name': '', 'openshift.io/requester': 'system:serviceaccount:kube-system:olminstall-cluster-admin', 'openshift.io/sa.scc.mcs': 's0:c48,c37', 'openshift.io/sa.scc.supplemental-groups': '1002330000/10000', 'openshift.io/sa.scc.uid-range': '1002330000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-07-09T16:03:02Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:03:02Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}, {'manager': 'openshift-apiserver', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:03:02Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/description': {}, 'f:openshift.io/display-name': {}, 'f:openshift.io/requester': {}}, 'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'openshift-controller-manager', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:03:02Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:finalizers': {}}}, 'subresource': 'finalize'}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Project] 2026-07-09T16:08:57.464328Z [info ] Wait until Project test-kserve-raw-token-authentication is deleted [ocp_resources Project] _ 4 of 5 completed, 1 Pass, 2 Fail, 0 Skip, 0 XPass, 0 XFail, 1 Error, 0 ReRun _ tests/model_serving/model_server/kserve/authentication/test_non_admin_users.py::TestRawUnprivilegedUser::test_non_admin_deploy_raw_and_query_model[unprivileged_model_namespace0-unprivileged_s3_ovms_raw_inference_service0]  - test_non_admin_deploy_raw_and_query_model[unprivileged_model_namespace0-unprivileged_s3_ovms_raw_inference_service0] - -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-07-09T16:09:05.616988Z [info ] Executing class fixture: unprivileged_model_namespace [conftest] 2026-07-09T16:09:05.617118Z [info ] kind: ProjectRequest api version: project.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:09:05.617228Z [info ] Create ProjectRequest test-non-admin-raw [ocp_resources ProjectRequest] 2026-07-09T16:09:05.617334Z [info ] Posting {'apiVersion': 'project.openshift.io/v1', 'kind': 'ProjectRequest', 'metadata': {'name': 'test-non-admin-raw'}} [ocp_resources ProjectRequest] 2026-07-09T16:09:05.725005Z [info ] kind: Project api version: project.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:09:05.725162Z [info ] Wait for Project test-non-admin-raw status to be Active [ocp_resources Project] 2026-07-09T16:09:05.743886Z [info ] Status of Project test-non-admin-raw is Active [ocp_resources Project] 2026-07-09T16:09:05.744040Z [info ] Executing class fixture: http_s3_ovms_serving_runtime [conftest] 2026-07-09T16:09:05.744150Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:09:05.776148Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:09:05.776305Z [info ] Create ServingRuntime http-mnist-runtime [ocp_resources ServingRuntime] 2026-07-09T16:09:05.776460Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'http-mnist-runtime', 'namespace': 'test-non-admin-raw'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:5bb57ef927e7481cfc768993faa8e1e70b27154ecc81a88991bd5428001c1ea6', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-07-09T16:09:05.806660Z [info ] Executing class fixture: unprivileged_ci_endpoint_s3_secret [conftest] 2026-07-09T16:09:05.820766Z [info ] Create Secret ci-bucket-unprivileged [ocp_resources Secret] 2026-07-09T16:09:05.821029Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'ci-bucket-unprivileged', 'labels': {'opendatahub.io/managed': 'true', 'opendatahub.io/dashboard': 'true'}, 'annotations': {'opendatahub.io/connection-type': 's3', 'serving.kserve.io/s3-endpoint': 's3.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-1', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-verifyssl': '0', 'serving.kserve.io/s3-usehttps': '1'}, 'namespace': 'test-non-admin-raw'}, 'data': '*******'} [ocp_resources Secret] 2026-07-09T16:09:05.843875Z [info ] Wait until Secret ci-bucket-unprivileged is created [ocp_resources Secret] 2026-07-09T16:09:05.857592Z [info ] Executing class fixture: unprivileged_s3_ovms_raw_inference_service [conftest] 2026-07-09T16:09:05.869238Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-07-09T16:09:05.869395Z [info ] Create InferenceService http-onnx-raw [ocp_resources InferenceService] 2026-07-09T16:09:05.869567Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'http-onnx-raw', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'test-non-admin-raw'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'openvino_ir'}, 'version': None, 'runtime': 'http-mnist-runtime', 'storage': {'key': 'ci-bucket-unprivileged', 'path': '/test-dir/'}}}}} [ocp_resources InferenceService] 2026-07-09T16:09:05.904158Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-07-09T16:09:06.926434Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-07-09T16:09:06.974187Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-07-09T16:09:27.182196Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-07-09T16:09:27.247869Z [info ] Wait for Deployment http-onnx-raw-predictor to be deployed: True [ocp_resources Deployment] 2026-07-09T16:09:27.281174Z [info ] Wait for InferenceService/http-onnx-raw's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-07-09T16:09:27.281377Z [info ] Wait until InferenceService http-onnx-raw is created [ocp_resources InferenceService] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-07-09T16:09:27.415244Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:09:27.448746Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:09:27.454818Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:09:27.554797Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:09:27.581228Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils [2026-07-09T16:09:27Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed 2026-07-09T16:09:27.878265Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils FAILED TEST: TestRawUnprivilegedUser.test_non_admin_deploy_raw_and_query_model[unprivileged_model_namespace0-unprivileged_s3_ovms_raw_inference_service0] STATUS: FAILED 2026-07-09T16:09:27.883182Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_non_admin_users.py", line 37, in test_non_admin_deploy_raw_and_query_model verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=unprivileged_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... use_default_query=True, ^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx-raw","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} [conftest] ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-07-09T16:09:27.883457Z [info ] Delete InferenceService http-onnx-raw [ocp_resources InferenceService] 2026-07-09T16:09:27.922146Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'openvino_ir', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-07-09T16:09:05Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storage': {'.': {}, 'f:key': {}, 'f:path': {}}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-07-09T16:09:05Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-07-09T16:09:06Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-07-09T16:09:18Z'}], 'name': 'http-onnx-raw', 'namespace': 'test-non-admin-raw', 'resourceVersion': '385935889', 'uid': 'f81f89ca-9a18-4f90-a77c-bfaeb0020b02'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'openvino_ir'}, 'name': '', 'resources': {}, 'runtime': 'http-mnist-runtime', 'storage': {'key': 'ci-bucket-unprivileged', 'path': '/test-dir/'}}}}, 'status': {'address': {'url': 'http://http-onnx-raw-predictor.test-non-admin-raw.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://http-onnx-raw-predictor-test-non-admin-raw.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-07-09T16:09:06Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-07-09T16:09:18Z', 'message': 'ReplicaSet "http-onnx-raw-predictor-6fbd98c4f" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-07-09T16:09:18Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T16:09:06Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'http-mnist-runtime', 'url': 'http://http-onnx-raw-predictor.test-non-admin-raw.svc.cluster.local'}} [ocp_resources InferenceService] 2026-07-09T16:09:27.941509Z [info ] Wait until InferenceService http-onnx-raw is deleted [ocp_resources InferenceService] 2026-07-09T16:09:28.973443Z [info ] Delete Secret ci-bucket-unprivileged [ocp_resources Secret] 2026-07-09T16:09:29.013089Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'ci-bucket-unprivileged', 'namespace': 'test-non-admin-raw', 'uid': '2f005d07-a897-4968-bbbd-827d9c7e9474', 'resourceVersion': '385935604', 'creationTimestamp': '2026-07-09T16:09:05Z', 'labels': {'opendatahub.io/dashboard': 'true', 'opendatahub.io/managed': 'true'}, 'annotations': {'opendatahub.io/connection-type': 's3', 'serving.kserve.io/s3-endpoint': 's3.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-1', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-usehttps': '1', 'serving.kserve.io/s3-verifyssl': '0'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:09:05Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:AWS_ACCESS_KEY_ID': {}, 'f:AWS_DEFAULT_REGION': {}, 'f:AWS_S3_BUCKET': {}, 'f:AWS_S3_ENDPOINT': {}, 'f:AWS_SECRET_ACCESS_KEY': {}}, 'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/connection-type': {}, 'f:serving.kserve.io/s3-endpoint': {}, 'f:serving.kserve.io/s3-region': {}, 'f:serving.kserve.io/s3-useanoncredential': {}, 'f:serving.kserve.io/s3-usehttps': {}, 'f:serving.kserve.io/s3-verifyssl': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}, 'f:opendatahub.io/managed': {}}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-09T16:09:29.030839Z [info ] Wait until Secret ci-bucket-unprivileged is deleted [ocp_resources Secret] 2026-07-09T16:09:29.047987Z [info ] Delete ServingRuntime http-mnist-runtime [ocp_resources ServingRuntime] 2026-07-09T16:09:29.077114Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'creationTimestamp': '2026-07-09T16:09:05Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/recommended-accelerators': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-07-09T16:09:05Z'}], 'name': 'http-mnist-runtime', 'namespace': 'test-non-admin-raw', 'resourceVersion': '385935603', 'uid': 'f702ac33-7900-4fb2-8730-b022a8d0757f'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:5bb57ef927e7481cfc768993faa8e1e70b27154ecc81a88991bd5428001c1ea6', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-07-09T16:09:29.096323Z [info ] Wait until ServingRuntime http-mnist-runtime is deleted [ocp_resources ServingRuntime] 2026-07-09T16:09:29.201427Z [info ] Delete Project test-non-admin-raw [ocp_resources Project] 2026-07-09T16:09:29.254342Z [info ] Deleting {'kind': 'Project', 'apiVersion': 'project.openshift.io/v1', 'metadata': {'name': 'test-non-admin-raw', 'uid': 'a1dae18c-675d-409c-9c4a-95d7938f0e14', 'resourceVersion': '385935591', 'creationTimestamp': '2026-07-09T16:09:05Z', 'labels': {'kubernetes.io/metadata.name': 'test-non-admin-raw', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/description': '', 'openshift.io/display-name': '', 'openshift.io/requester': 'system:serviceaccount:kube-system:olminstall-cluster-admin', 'openshift.io/sa.scc.mcs': 's0:c49,c14', 'openshift.io/sa.scc.supplemental-groups': '1002380000/10000', 'openshift.io/sa.scc.uid-range': '1002380000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-07-09T16:09:05Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:09:05Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}, {'manager': 'openshift-apiserver', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:09:05Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/description': {}, 'f:openshift.io/display-name': {}, 'f:openshift.io/requester': {}}, 'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'openshift-controller-manager', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:09:05Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:finalizers': {}}}, 'subresource': 'finalize'}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Project] 2026-07-09T16:09:29.531789Z [info ] Wait until Project test-non-admin-raw is deleted [ocp_resources Project] _ TestRawUnprivilegedUser.test_non_admin_deploy_raw_and_query_model[unprivileged_model_namespace0-unprivileged_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_non_admin_users.py", line 37, in test_non_admin_deploy_raw_and_query_model verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=unprivileged_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... use_default_query=True, ^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx-raw","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} _ 5 of 5 completed, 1 Pass, 3 Fail, 0 Skip, 0 XPass, 0 XFail, 1 Error, 0 ReRun _ = 3 failed, 1 passed, 219 deselected, 728 warnings, 1 error in 418.53s (0:06:58) = ==================================== ERRORS ==================================== _ ERROR at setup of TestKserveTokenAuthenticationRawForRest.test_disabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 170, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 536, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1710, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 806, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 630, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1232, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 924, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1328, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 998, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/conftest.py", line 100, in patched_remove_raw_authentication_isvc wait_for_raw_isvc_https_infer_ready(isvc=http_s3_ovms_raw_inference_service, token=None) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 234, in wait_for_raw_isvc_https_infer_ready for ok in TimeoutSampler(wait_timeout=timeout, sleep=sleep, func=_https_infer_ok): ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 300 Function: tests.model_serving.model_server.utils._https_infer_ok Last exception: N/A. =================================== FAILURES =================================== _ TestKserveTokenAuthenticationRawForRest.test_model_authentication_using_rest_raw[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py", line 39, in test_model_authentication_using_rest_raw verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=http_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... token=http_raw_inference_token, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} _ TestKserveTokenAuthenticationRawForRest.test_re_enabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py", line 61, in test_re_enabled_raw_model_authentication verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=http_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... token=http_raw_inference_token, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} _ TestRawUnprivilegedUser.test_non_admin_deploy_raw_and_query_model[unprivileged_model_namespace0-unprivileged_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_non_admin_users.py", line 37, in test_non_admin_deploy_raw_and_query_model verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=unprivileged_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... use_default_query=True, ^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx-raw","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} =============================== warnings summary =============================== tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py: 645 warnings tests/model_serving/model_server/kserve/authentication/test_non_admin_users.py: 83 warnings /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-07.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: /workspace/tests-shared/tests-payload/results/model_server-smoke.xml - =========================== short test summary info ============================ ERROR tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_disabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] - timeout_sampler.TimeoutExpiredError: Timed Out: 300 Function: tests.model_serving.model_server.utils._https_infer_ok Last exception: N/A. FAILED tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_model_authentication_using_rest_raw[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] - AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} FAILED tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_re_enabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] - AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} FAILED tests/model_serving/model_server/kserve/authentication/test_non_admin_users.py::TestRawUnprivilegedUser::test_non_admin_deploy_raw_and_query_model[unprivileged_model_namespace0-unprivileged_s3_ovms_raw_inference_service0] - AssertionError: Expected: {"model_name":"http-onnx-raw","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} = 3 failed, 1 passed, 219 deselected, 728 warnings, 1 error in 418.53s (0:06:58) =