--- BVT pytest attempt 2026-07-09T15:55:26.450991+00:00 (/.local/bin/uv --directory /home/odh/opendatahub-tests...) --- 2026-07-09T15:55:28.516421Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-07-09T15:55:28.516935Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-07-09T15:55:28.718550Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-07-09T15:55:28.830735Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( ============================= test session starts ============================== platform linux -- Python 3.14.6, pytest-9.1.1, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python3 cachedir: .pytest_cache metadata: {'Python': '3.14.6', 'Platform': 'Linux-5.14.0-570.120.1.el9_6.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.1.1', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.14.0', 'langsmith': '0.9.0', 'asyncio': '1.4.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.5.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.3.2'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.14.0, langsmith-0.9.0, asyncio-1.4.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.5.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.3.2 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... collected 224 items / 219 deselected / 5 selected tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_model_authentication_using_rest_raw[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0]  ----- test_model_authentication_using_rest_raw[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] ----- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-07-09T15:55:29.302511Z [info ] Executing session fixture: admin_client [conftest] 2026-07-09T15:55:29.302649Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:55:29.305768Z [info ] Executing session fixture: dsc_resource [conftest] 2026-07-09T15:55:29.305914Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-07-09T15:55:29.368567Z [info ] Executing package fixture: enabled_kserve_in_dsc [conftest] 2026-07-09T15:55:29.385119Z [warning ] Component kserve was already set to managementState Managed [utilities.data_science_cluster_utils] name=utilities.data_science_cluster_utils 2026-07-09T15:55:29.386459Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-07-09T15:55:29.386620Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-07-09T15:55:29.386774Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-07-09T15:55:29.386923Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-07-09T15:55:29.387983Z [info ] Executing session fixture: os_path_environment [conftest] 2026-07-09T15:55:29.388263Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-07-09T15:55:29.388435Z [info ] Executing session fixture: bin_directory [conftest] 2026-07-09T15:55:29.389352Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-07-09T15:55:29.410832Z [info ] kind: ConsoleCLIDownload api version: console.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:55:29.437861Z [info ] All oc console cli download links: ['https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar'] [utilities.infra] name=utilities.infra 2026-07-09T15:55:29.437951Z [info ] Downloading archive using: url=https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar [utilities.infra] name=utilities.infra 2026-07-09T15:55:31.470684Z [info ] Extract the downloaded archive. [utilities.infra] name=utilities.infra 2026-07-09T15:55:33.445756Z [info ] Downloaded file: ['oc']  [utilities.infra] name=utilities.infra 2026-07-09T15:55:33.586166Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-07-09T15:55:33.582952Z [info ] OC binary path: /tmp/pytest-6kqhfZv56s8rWXeoHwdt32/bin0/oc [tests.conftest] name=tests.conftest 2026-07-09T15:55:33.583062Z [info ] Adding /tmp/pytest-6kqhfZv56s8rWXeoHwdt32/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-07-09T15:55:33.586620Z [info ] Executing session fixture: nodes [conftest] 2026-07-09T15:55:34.543264Z [info ] Executing session fixture: dsci_resource [conftest] 2026-07-09T15:55:34.543434Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-07-09T15:55:34.559768Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-07-09T15:55:34.559927Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-07-09T15:55:34.560095Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-07-09T15:55:34.559068Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-07-09T15:55:34.560331Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-07-09T15:55:34.659194Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-07-09T15:55:34.751704Z [warning ] Skipping RHOAI resource checks, got --cluster-sanity-skip-rhoai-check [utilities.infra] name=utilities.infra 2026-07-09T15:55:34.753243Z [info ] Executing session fixture: autouse_fixtures [conftest] 2026-07-09T15:55:34.753383Z [info ] Executing session fixture: kserve_health_check [conftest] 2026-07-09T15:55:34.783143Z [info ] kind: Deployment api version: apps/v1 [ocp_resources.resource] 2026-07-09T15:55:34.839566Z [info ] KServe component health check passed [tests.model_serving.model_server.kserve.conftest] 2026-07-09T15:55:34.839743Z [info ] Executing session fixture: pytestconfig [conftest] 2026-07-09T15:55:34.839891Z [info ] Executing session fixture: aws_access_key_id [conftest] 2026-07-09T15:55:34.840159Z [info ] Executing session fixture: aws_secret_access_key [conftest] 2026-07-09T15:55:34.840286Z [info ] Executing session fixture: ci_s3_bucket_endpoint [conftest] 2026-07-09T15:55:34.840395Z [info ] Executing session fixture: valid_aws_config [conftest] 2026-07-09T15:55:34.889108Z [info ] AWS credentials validated successfully via STS GetCallerIdentity [tests.conftest] name=tests.conftest 2026-07-09T15:55:34.891014Z [info ] Executing session fixture: use_unprivileged_client [conftest] 2026-07-09T15:55:34.891201Z [info ] Executing session fixture: kubconfig_filepath [conftest] 2026-07-09T15:55:34.891319Z [info ] Executing session fixture: cluster_authentication [conftest] 2026-07-09T15:55:34.929465Z [info ] kind: Authentication api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:55:34.945411Z [info ] Executing session fixture: is_byoidc [conftest] 2026-07-09T15:55:34.960485Z [info ] Executing session fixture: non_admin_user_password [conftest] 2026-07-09T15:55:34.960633Z [info ] Executing session fixture: unprivileged_client [conftest] 2026-07-09T15:55:34.957933Z [warning ] Unprivileged client is not enabled, using admin client [tests.conftest] name=tests.conftest 2026-07-09T15:55:34.960921Z [info ] Executing session fixture: teardown_resources [conftest] 2026-07-09T15:55:34.961127Z [info ] Executing session fixture: ci_s3_bucket_name [conftest] 2026-07-09T15:55:34.961254Z [info ] Executing session fixture: ci_s3_bucket_region [conftest] 2026-07-09T15:55:34.961358Z [info ] Executing session fixture: models_s3_bucket_name [conftest] 2026-07-09T15:55:34.961470Z [info ] Executing session fixture: models_s3_bucket_region [conftest] 2026-07-09T15:55:34.961592Z [info ] Executing session fixture: models_s3_bucket_endpoint [conftest] 2026-07-09T15:55:34.961698Z [info ] Executing class fixture: unprivileged_model_namespace [conftest] 2026-07-09T15:55:34.961803Z [info ] kind: ProjectRequest api version: project.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:55:34.961913Z [info ] Create ProjectRequest test-kserve-raw-token-authentication [ocp_resources ProjectRequest] 2026-07-09T15:55:34.962017Z [info ] Posting {'apiVersion': 'project.openshift.io/v1', 'kind': 'ProjectRequest', 'metadata': {'name': 'test-kserve-raw-token-authentication'}} [ocp_resources ProjectRequest] 2026-07-09T15:55:35.120462Z [info ] kind: Project api version: project.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:55:35.120743Z [info ] Wait for Project test-kserve-raw-token-authentication status to be Active [ocp_resources Project] 2026-07-09T15:55:35.139578Z [info ] Status of Project test-kserve-raw-token-authentication is Active [ocp_resources Project] 2026-07-09T15:55:35.139750Z [info ] Executing class fixture: http_s3_ovms_serving_runtime [conftest] 2026-07-09T15:55:35.139911Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:55:35.225649Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:55:35.225847Z [info ] Create ServingRuntime http-mnist-runtime [ocp_resources ServingRuntime] 2026-07-09T15:55:35.226157Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'http-mnist-runtime', 'namespace': 'test-kserve-raw-token-authentication'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-07-09T15:55:35.271787Z [info ] Executing class fixture: ci_endpoint_s3_secret [conftest] 2026-07-09T15:55:35.288165Z [info ] Create Secret ci-bucket-secret [ocp_resources Secret] 2026-07-09T15:55:35.288368Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'ci-bucket-secret', 'labels': {'opendatahub.io/managed': 'true', 'opendatahub.io/dashboard': 'true'}, 'annotations': {'opendatahub.io/connection-type': 's3', 'serving.kserve.io/s3-endpoint': 's3.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-1', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-verifyssl': '0', 'serving.kserve.io/s3-usehttps': '1'}, 'namespace': 'test-kserve-raw-token-authentication'}, 'data': '*******'} [ocp_resources Secret] 2026-07-09T15:55:35.315947Z [info ] Wait until Secret ci-bucket-secret is created [ocp_resources Secret] 2026-07-09T15:55:35.329155Z [info ] Executing class fixture: models_endpoint_s3_secret [conftest] 2026-07-09T15:55:35.344901Z [info ] Create Secret models-bucket-secret [ocp_resources Secret] 2026-07-09T15:55:35.345276Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'models-bucket-secret', 'labels': {'opendatahub.io/managed': 'true', 'opendatahub.io/dashboard': 'true'}, 'annotations': {'opendatahub.io/connection-type': 's3', 'serving.kserve.io/s3-endpoint': 's3.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-1', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-verifyssl': '0', 'serving.kserve.io/s3-usehttps': '1'}, 'namespace': 'test-kserve-raw-token-authentication'}, 'data': '*******'} [ocp_resources Secret] 2026-07-09T15:55:35.371429Z [info ] Wait until Secret models-bucket-secret is created [ocp_resources Secret] 2026-07-09T15:55:35.388015Z [info ] Executing class fixture: model_service_account [conftest] 2026-07-09T15:55:35.388199Z [info ] Create ServiceAccount models-bucket-sa [ocp_resources ServiceAccount] 2026-07-09T15:55:35.388317Z [info ] Posting {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'models-bucket-sa', 'namespace': 'test-kserve-raw-token-authentication'}, 'secrets': [{'name': 'models-bucket-secret'}]} [ocp_resources ServiceAccount] 2026-07-09T15:55:35.415991Z [info ] Executing class fixture: http_s3_ovms_raw_inference_service [conftest] 2026-07-09T15:55:35.430935Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-07-09T15:55:35.431275Z [info ] Create InferenceService http-onnx [ocp_resources InferenceService] 2026-07-09T15:55:35.431904Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'http-onnx', 'labels': {'networking.kserve.io/visibility': 'exposed'}, 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment', 'security.opendatahub.io/enable-auth': 'true'}, 'namespace': 'test-kserve-raw-token-authentication'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'openvino_ir'}, 'version': None, 'runtime': 'http-mnist-runtime', 'storage': {'key': 'ci-bucket-secret', 'path': '/test-dir/'}}, 'serviceAccountName': 'models-bucket-sa'}}} [ocp_resources InferenceService] 2026-07-09T15:55:35.508565Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-07-09T15:55:36.533747Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-07-09T15:55:36.586223Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-07-09T15:56:16.976499Z [info ] kind: Deployment api version: apps/v1 [ocp_resources.resource] 2026-07-09T15:56:16.990328Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-07-09T15:56:17.048289Z [info ] Wait for Deployment http-onnx-predictor to be deployed: True [ocp_resources Deployment] 2026-07-09T15:56:17.074501Z [info ] Wait for InferenceService/http-onnx's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-07-09T15:56:17.074889Z [info ] Wait until InferenceService http-onnx is created [ocp_resources InferenceService] 2026-07-09T15:56:17.146452Z [info ] Executing class fixture: http_raw_view_role [conftest] 2026-07-09T15:56:17.146638Z [info ] kind: Role api version: rbac.authorization.k8s.io/v1 [ocp_resources.resource] 2026-07-09T15:56:17.146762Z [info ] Create Role http-onnx-view  [ocp_resources Role] 2026-07-09T15:56:17.146897Z [info ] Posting {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'Role', 'metadata': {'name': 'http-onnx-view', 'namespace': 'test-kserve-raw-token-authentication'}, 'rules': [{'apiGroups': ['serving.kserve.io'], 'resources': ['inferenceservices'], 'verbs': ['get'], 'resourceNames': ['http-onnx']}]} [ocp_resources Role] 2026-07-09T15:56:17.174987Z [info ] Executing class fixture: http_raw_role_binding [conftest] 2026-07-09T15:56:17.175281Z [info ] kind: RoleBinding api version: rbac.authorization.k8s.io/v1 [ocp_resources.resource] 2026-07-09T15:56:17.175492Z [info ] Create RoleBinding http-models-bucket-sa-view [ocp_resources RoleBinding] 2026-07-09T15:56:17.175623Z [info ] Posting {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'RoleBinding', 'metadata': {'name': 'http-models-bucket-sa-view', 'namespace': 'test-kserve-raw-token-authentication'}, 'subjects': [{'kind': 'ServiceAccount', 'name': 'models-bucket-sa'}], 'roleRef': {'kind': 'Role', 'name': 'http-onnx-view', 'apiGroup': 'rbac.authorization.k8s.io'}} [ocp_resources RoleBinding] 2026-07-09T15:56:17.204436Z [info ] Executing class fixture: http_raw_inference_token [conftest] 2026-07-09T15:56:17.204608Z [info ] Running oc create token -n test-kserve-raw-token-authentication models-bucket-sa command [pyhelper_utils.shell] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-07-09T15:56:17.377142Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:56:17.416657Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:56:17.423497Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:17.517496Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:17.529696Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:56:17.544690Z [info ] Running Hide By User command  [pyhelper_utils.shell] 2026-07-09T15:56:17.617983Z [info ] Inference output: HTTP/1.1 404 Not Found access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * content-length: 53 content-type: application/json; charset=utf-8 date: Thu, 09 Jul 2026 15:56:17 GMT set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=a79568e0d33dfb56; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils FAILED _ TestKserveTokenAuthenticationRawForRest.test_model_authentication_using_rest_raw[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py", line 39, in test_model_authentication_using_rest_raw verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=http_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... token=http_raw_inference_token, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} _ 1 of 5 completed, 0 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_disabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0]  TEST: TestKserveTokenAuthenticationRawForRest.test_model_authentication_using_rest_raw[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] STATUS: FAILED 2026-07-09T15:56:17.632259Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py", line 39, in test_model_authentication_using_rest_raw verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=http_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... token=http_raw_inference_token, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} [conftest] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ------ test_disabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] ------ -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-07-09T15:56:17.632823Z [info ] Executing function fixture: patched_remove_raw_authentication_isvc [conftest] 2026-07-09T15:56:17.632944Z [info ] ResourceEdit: Backing up old data [ocp_resources.resource] 2026-07-09T15:56:17.649974Z [info ] ResourceEdits: Updating data for resource InferenceService http-onnx [ocp_resources.resource] 2026-07-09T15:56:17.650159Z [info ] Update InferenceService http-onnx: {'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, 'name': 'http-onnx'}} [ocp_resources InferenceService] 2026-07-09T15:56:17.681005Z [info ] Wait for InferenceService/http-onnx's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-07-09T15:56:17.681287Z [info ] Wait until InferenceService http-onnx is created [ocp_resources InferenceService] 2026-07-09T15:56:17.751666Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-07-09T15:56:17.868489Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:56:17.913182Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:56:17.919665Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:17.990147Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:18.004323Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:56:18.062146Z [info ] Inference output: HTTP/1.0 400 Bad Request set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=57637ca4eb820db0e541fc5c99dc72a0; path=/; HttpOnly; Secure; SameSite=None Client sent an HTTP request to an HTTPS server.  [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:56:23.092741Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:56:23.132972Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:56:23.139656Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:23.221599Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:23.235011Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:56:23.309450Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=d3d0fc594cd1eaca7b59a774680639ed; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:56:28.340488Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:56:28.382152Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:56:28.386569Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:28.455194Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:28.471663Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:56:28.540517Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=d3d0fc594cd1eaca7b59a774680639ed; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:56:33.568993Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:56:33.607576Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:56:33.770409Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:33.840062Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:33.853434Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:56:33.915839Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=d3d0fc594cd1eaca7b59a774680639ed; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:56:38.943817Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:56:38.982214Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:56:38.988451Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:39.070451Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:39.084146Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:56:39.149389Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=d3d0fc594cd1eaca7b59a774680639ed; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:56:44.177091Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:56:44.220043Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:56:44.224555Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:44.295862Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:44.309402Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:56:44.370210Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=d3d0fc594cd1eaca7b59a774680639ed; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:56:49.399698Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:56:49.450469Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:56:49.456804Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:49.529230Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:49.543281Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:56:49.607070Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=d3d0fc594cd1eaca7b59a774680639ed; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:56:54.634574Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:56:54.671145Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:56:54.677483Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:54.748844Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:54.766065Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:56:54.829282Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=d3d0fc594cd1eaca7b59a774680639ed; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:56:59.856467Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:56:59.900431Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:56:59.908490Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:59.976250Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:56:59.991710Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:57:00.048924Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:57:05.078563Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:57:05.117082Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:57:05.127311Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:05.192370Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:05.204729Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:57:05.271501Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:57:10.298514Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:57:10.337717Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:57:10.344486Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:10.412826Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:10.425661Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:57:10.486669Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:57:15.513760Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:57:15.554441Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:57:15.560769Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:15.637549Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:15.657870Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:57:15.724001Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:57:20.751638Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:57:20.804491Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:57:20.816188Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:20.886119Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:20.900258Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:57:20.965956Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:57:25.993951Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:57:26.034994Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:57:26.041557Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:26.109077Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:26.123687Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:57:26.193325Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:57:31.230828Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:57:31.288913Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:57:31.295481Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:31.382459Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:31.395299Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:57:31.488121Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:57:36.516083Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:57:36.553202Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:57:36.562630Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:36.631800Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:36.646387Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:57:36.705225Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:57:41.955729Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:57:41.994692Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:57:42.001244Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:42.079143Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:42.095901Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:57:42.160761Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:57:47.188529Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:57:47.227225Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:57:47.233843Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:47.297040Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:47.309622Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:57:47.376138Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:57:52.410949Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:57:52.450592Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:57:52.457320Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:52.540961Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:52.553785Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:57:52.613769Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:57:57.648722Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:57:57.710732Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:57:57.715367Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:57.791551Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:57:57.813568Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:57:57.898892Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:58:02.930248Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:58:02.980052Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:58:02.984708Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:03.070275Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:03.084434Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:58:03.162569Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:58:08.197110Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:58:08.252341Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:58:08.262818Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:08.327719Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:08.357889Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:58:08.442170Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:58:13.473926Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:58:13.522322Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:58:13.528965Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:13.618178Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:13.635185Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:58:13.700961Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:58:18.728974Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:58:18.767596Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:58:18.774374Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:18.884771Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:18.911059Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:58:19.010762Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:58:24.039882Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:58:24.085610Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:58:24.090314Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:24.158348Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:24.172270Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:58:24.254082Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:58:29.298425Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:58:29.451833Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:58:29.458462Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:29.520208Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:29.532681Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:58:29.605886Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:58:34.634585Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:58:34.671770Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:58:34.678374Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:34.749370Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:34.762875Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:58:34.855628Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:58:39.882565Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:58:39.931759Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:58:39.941149Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:40.011219Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:40.026207Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:58:40.099336Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:58:45.131009Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:58:45.171172Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:58:45.177491Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:45.244328Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:45.260503Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:58:45.326567Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:58:50.355571Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:58:50.404714Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:58:50.411376Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:50.477962Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:50.494448Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:58:50.563110Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:58:55.589848Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:58:55.627731Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:58:55.634346Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:55.715059Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:58:55.728722Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:58:55.794179Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:59:00.822634Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:59:00.860010Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:59:00.866789Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:00.936291Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:00.951246Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:59:01.008507Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:59:06.035456Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:59:06.072924Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:59:06.079551Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:06.149688Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:06.163225Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:59:06.236219Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:59:11.263653Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:59:11.302805Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:59:11.314260Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:11.382887Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:11.396592Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:59:11.463585Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:59:16.491153Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:59:16.530358Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:59:16.536955Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:16.606864Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:16.619619Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:59:16.687612Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:59:21.714102Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:59:21.753068Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:59:21.759296Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:21.835774Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:21.847968Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:59:21.917959Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:59:26.945863Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:59:26.983921Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:59:26.990347Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:27.063411Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:27.078258Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:59:27.154350Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:59:32.182573Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:59:32.221568Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:59:32.228066Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:32.314160Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:32.327617Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:59:32.390018Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:59:37.418771Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:59:37.466477Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:59:37.471071Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:37.539649Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:37.553632Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:59:37.613403Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:59:42.641282Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:59:42.684182Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:59:42.693335Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:42.766193Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:42.779377Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:59:42.836949Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:59:47.886630Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:59:47.965232Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:59:47.969850Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:48.036935Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:48.053630Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:59:48.129422Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:59:53.168733Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:59:53.209757Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:59:53.215968Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:53.285017Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:53.301752Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:59:53.363322Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T15:59:58.417690Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T15:59:58.456162Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T15:59:58.462533Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:58.537548Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T15:59:58.551730Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T15:59:58.617391Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:00:03.658418Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:00:03.699960Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:00:03.704518Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:03.772771Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:03.786185Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:00:03.848296Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:00:08.878535Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:00:08.918130Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:00:08.922445Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:09.008067Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:09.021265Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:00:09.086410Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:00:14.126649Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:00:14.168895Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:00:14.178592Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:14.248856Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:14.263299Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:00:14.325283Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:00:19.365787Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:00:19.405302Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:00:19.411587Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:19.477640Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:19.491378Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:00:19.561438Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:00:24.592788Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:00:24.632198Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:00:24.639577Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:24.713128Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:24.728650Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:00:24.785431Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:00:29.815423Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:00:29.879100Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:00:29.885455Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:29.954253Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:29.968466Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:00:30.030845Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:00:35.064425Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:00:35.105156Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:00:35.111377Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:35.184412Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:35.198218Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:00:35.258091Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:00:40.286797Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:00:40.324523Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:00:40.330928Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:40.405443Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:40.419209Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:00:40.482583Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:00:45.514620Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:00:45.552511Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:00:45.562333Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:45.635851Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:45.649447Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:00:45.729641Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:00:50.758954Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:00:50.800075Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:00:50.807110Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:50.881313Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:50.894351Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:00:50.958478Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:00:55.987340Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:00:56.025065Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:00:56.031705Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:56.101909Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:00:56.115076Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:00:56.174610Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:01:01.203894Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:01:01.242300Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:01:01.249732Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:01.323514Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:01.336834Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:01:01.407324Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:01:06.435798Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:01:06.481546Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:01:06.481854Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:06.558555Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:06.572429Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:01:06.638257Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:01:11.666195Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:01:11.703441Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:01:11.709973Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:11.781823Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:11.795200Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:01:11.850349Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:01:16.879131Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:01:16.917709Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:01:16.927109Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:16.998925Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:17.013263Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:01:17.071751Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=7962deaf95ba69d7; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils 2026-07-09T16:01:22.072484Z [error ] Timeout: InferenceService http-onnx in test-kserve-raw-token-authentication external HTTPS infer not ready within 300s (token=no) [tests.model_serving.model_server.utils] name=tests.model_serving.model_server.utils 2026-07-09T16:01:22.075019Z [info ] ResourceEdits: Restoring data for resource InferenceService http-onnx [ocp_resources.resource] 2026-07-09T16:01:22.075334Z [info ] Update InferenceService http-onnx: {'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'true'}, 'name': 'http-onnx'}} [ocp_resources InferenceService] ERROR _ ERROR at setup of TestKserveTokenAuthenticationRawForRest.test_disabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 170, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 536, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1710, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 806, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 630, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1232, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 924, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1328, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 998, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/conftest.py", line 100, in patched_remove_raw_authentication_isvc wait_for_raw_isvc_https_infer_ready(isvc=http_s3_ovms_raw_inference_service, token=None) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 234, in wait_for_raw_isvc_https_infer_ready for ok in TimeoutSampler(wait_timeout=timeout, sleep=sleep, func=_https_infer_ok): ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 300 Function: tests.model_serving.model_server.utils._https_infer_ok Last exception: N/A. _ 2 of 5 completed, 0 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 1 Error, 0 ReRun _ tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_re_enabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0]  TEST: TestKserveTokenAuthenticationRawForRest.test_disabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] [setup] STATUS: ERROR 2026-07-09T16:01:22.132199Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 170, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 536, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1710, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 806, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 630, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1232, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 924, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1328, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 998, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/conftest.py", line 100, in patched_remove_raw_authentication_isvc wait_for_raw_isvc_https_infer_ready(isvc=http_s3_ovms_raw_inference_service, token=None) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 234, in wait_for_raw_isvc_https_infer_ready for ok in TimeoutSampler(wait_timeout=timeout, sleep=sleep, func=_https_infer_ok): ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 300 Function: tests.model_serving.model_server.utils._https_infer_ok Last exception: N/A. [conftest] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ----- test_re_enabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] ----- -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-07-09T16:01:22.157409Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:01:22.199900Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:01:22.206964Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:22.282140Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:22.300660Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils 2026-07-09T16:01:22.373940Z [info ] Inference output: HTTP/1.1 404 Not Found access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * content-length: 53 content-type: application/json; charset=utf-8 date: Thu, 09 Jul 2026 16:01:22 GMT set-cookie: 12d2eb568f824d2bcf01971f024ec5f9=c9bcd01e0a7b3118d1dcad317ab01727; path=/; HttpOnly; Secure; SameSite=None {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils FAILED _ TestKserveTokenAuthenticationRawForRest.test_re_enabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py", line 61, in test_re_enabled_raw_model_authentication verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=http_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... token=http_raw_inference_token, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} _ 3 of 5 completed, 0 Pass, 2 Fail, 0 Skip, 0 XPass, 0 XFail, 1 Error, 0 ReRun _ tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_cross_model_authentication_raw[http_s3_ovms_raw_inference_service_20-unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0]  TEST: TestKserveTokenAuthenticationRawForRest.test_re_enabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] STATUS: FAILED 2026-07-09T16:01:22.380919Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py", line 61, in test_re_enabled_raw_model_authentication verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=http_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... token=http_raw_inference_token, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} [conftest] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  test_cross_model_authentication_raw[http_s3_ovms_raw_inference_service_20-unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0]  -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-07-09T16:01:22.381619Z [info ] Executing class fixture: model_service_account_2 [conftest] 2026-07-09T16:01:22.381743Z [info ] Create ServiceAccount models-bucket-sa-2 [ocp_resources ServiceAccount] 2026-07-09T16:01:22.381859Z [info ] Posting {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'models-bucket-sa-2', 'namespace': 'test-kserve-raw-token-authentication'}, 'secrets': [{'name': 'models-bucket-secret'}]} [ocp_resources ServiceAccount] 2026-07-09T16:01:22.405898Z [info ] Executing class fixture: http_s3_ovms_raw_inference_service_2 [conftest] 2026-07-09T16:01:22.418810Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-07-09T16:01:22.418989Z [info ] Create InferenceService http-onnx-2 [ocp_resources InferenceService] 2026-07-09T16:01:22.419559Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'http-onnx-2', 'labels': {'networking.kserve.io/visibility': 'exposed'}, 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment', 'security.opendatahub.io/enable-auth': 'true'}, 'namespace': 'test-kserve-raw-token-authentication'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'openvino_ir'}, 'version': None, 'runtime': 'http-mnist-runtime', 'storage': {'key': 'ci-bucket-secret', 'path': '/test-dir/'}}, 'serviceAccountName': 'models-bucket-sa-2'}}} [ocp_resources InferenceService] 2026-07-09T16:01:22.464308Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-07-09T16:01:23.493388Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-07-09T16:01:23.543482Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-07-09T16:01:33.787963Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-07-09T16:01:33.840614Z [info ] Wait for Deployment http-onnx-2-predictor to be deployed: True [ocp_resources Deployment] 2026-07-09T16:01:33.868676Z [info ] Wait for InferenceService/http-onnx-2's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-07-09T16:01:33.868814Z [info ] Wait until InferenceService http-onnx-2 is created [ocp_resources InferenceService] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-07-09T16:01:33.971347Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:01:34.017256Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:01:34.022005Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:34.105674Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:34.140955Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils PASSED2026-07-09T16:01:34.251527Z [info ] Inference output: HTTP/1.1 403 Forbidden content-type: text/plain; charset=utf-8 x-content-type-options: nosniff date: Thu, 09 Jul 2026 16:01:34 GMT content-length: 145 set-cookie: 69f764f782e5acf3035b89de67f2b2b6=6b99db2e9ca646ba; path=/; HttpOnly; Secure; SameSite=None Forbidden (user=system:serviceaccount:test-kserve-raw-token-authentication:models-bucket-sa, verb=get, resource=inferenceservices, subresource=)  [utilities.inference_utils] name=utilities.inference_utils  TEST: TestKserveTokenAuthenticationRawForRest.test_cross_model_authentication_raw[http_s3_ovms_raw_inference_service_20-unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-07-09T16:01:34.253606Z [info ] Delete InferenceService http-onnx-2 [ocp_resources InferenceService] 2026-07-09T16:01:34.299875Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'openvino_ir', 'security.opendatahub.io/enable-auth': 'true', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-07-09T16:01:22Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'labels': {'networking.kserve.io/visibility': 'exposed'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:security.opendatahub.io/enable-auth': {}, 'f:serving.kserve.io/deploymentMode': {}}, 'f:labels': {'.': {}, 'f:networking.kserve.io/visibility': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storage': {'.': {}, 'f:key': {}, 'f:path': {}}}, 'f:serviceAccountName': {}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-07-09T16:01:22Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-07-09T16:01:22Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-07-09T16:01:33Z'}], 'name': 'http-onnx-2', 'namespace': 'test-kserve-raw-token-authentication', 'resourceVersion': '250884243', 'uid': '0b873679-7808-44ac-baae-ac838839e3e3'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'openvino_ir'}, 'name': '', 'resources': {}, 'runtime': 'http-mnist-runtime', 'storage': {'key': 'ci-bucket-secret', 'path': '/test-dir/'}}, 'serviceAccountName': 'models-bucket-sa-2'}}, 'status': {'address': {'url': 'https://http-onnx-2-predictor.test-kserve-raw-token-authentication.svc.cluster.local:8443'}, 'components': {'predictor': {'url': 'http://http-onnx-2-predictor-test-kserve-raw-token-authentication.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-07-09T16:01:22Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-07-09T16:01:33Z', 'message': 'ReplicaSet "http-onnx-2-predictor-774cb48ff" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-07-09T16:01:33Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T16:01:22Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'http-mnist-runtime', 'url': 'https://http-onnx-2-test-kserve-raw-token-authentication.apps.ods-qe-psi-23.osp.rh-ods.com'}} [ocp_resources InferenceService] 2026-07-09T16:01:34.320447Z [info ] Wait until InferenceService http-onnx-2 is deleted [ocp_resources InferenceService] 2026-07-09T16:01:35.350266Z [info ] Delete ServiceAccount models-bucket-sa-2 [ocp_resources ServiceAccount] 2026-07-09T16:01:35.378769Z [info ] Deleting {'kind': 'ServiceAccount', 'apiVersion': 'v1', 'metadata': {'name': 'models-bucket-sa-2', 'namespace': 'test-kserve-raw-token-authentication', 'uid': '4ea59f1f-0d13-4256-b817-a3b538fdcb35', 'resourceVersion': '250883894', 'creationTimestamp': '2026-07-09T16:01:22Z', 'annotations': {'openshift.io/internal-registry-pull-secret-ref': 'models-bucket-sa-2-dockercfg-cbwn8'}, 'managedFields': [{'manager': 'openshift.io/image-registry-pull-secrets_service-account-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-07-09T16:01:22Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:imagePullSecrets': {}, 'f:metadata': {'f:annotations': {'f:openshift.io/internal-registry-pull-secret-ref': {}}}, 'f:secrets': {'k:{"name":"models-bucket-sa-2-dockercfg-cbwn8"}': {}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:01:22Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:secrets': {'.': {}, 'k:{"name":"models-bucket-secret"}': {}}}}]}, 'secrets': [{'name': 'models-bucket-secret'}, {'name': 'models-bucket-sa-2-dockercfg-cbwn8'}], 'imagePullSecrets': [{'name': 'models-bucket-sa-2-dockercfg-cbwn8'}]} [ocp_resources ServiceAccount] 2026-07-09T16:01:35.394281Z [info ] Wait until ServiceAccount models-bucket-sa-2 is deleted [ocp_resources ServiceAccount] 2026-07-09T16:01:35.407696Z [info ] Delete RoleBinding http-models-bucket-sa-view [ocp_resources RoleBinding] 2026-07-09T16:01:35.650547Z [info ] Deleting {'kind': 'RoleBinding', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'metadata': {'name': 'http-models-bucket-sa-view', 'namespace': 'test-kserve-raw-token-authentication', 'uid': 'ec650d72-d150-4dde-8d17-016ff24e15d7', 'resourceVersion': '250876124', 'creationTimestamp': '2026-07-09T15:56:17Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'time': '2026-07-09T15:56:17Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:roleRef': {}, 'f:subjects': {}}}]}, 'subjects': [{'kind': 'ServiceAccount', 'name': 'models-bucket-sa'}], 'roleRef': {'apiGroup': 'rbac.authorization.k8s.io', 'kind': 'Role', 'name': 'http-onnx-view'}} [ocp_resources RoleBinding] 2026-07-09T16:01:35.667492Z [info ] Wait until RoleBinding http-models-bucket-sa-view is deleted [ocp_resources RoleBinding] 2026-07-09T16:01:35.680557Z [info ] Delete Role http-onnx-view  [ocp_resources Role] 2026-07-09T16:01:35.706157Z [info ] Deleting {'kind': 'Role', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'metadata': {'name': 'http-onnx-view', 'namespace': 'test-kserve-raw-token-authentication', 'uid': '150fdbd1-0bf9-4155-9c4e-9b66fed25784', 'resourceVersion': '250876123', 'creationTimestamp': '2026-07-09T15:56:17Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'time': '2026-07-09T15:56:17Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:rules': {}}}]}, 'rules': [{'verbs': ['get'], 'apiGroups': ['serving.kserve.io'], 'resources': ['inferenceservices'], 'resourceNames': ['http-onnx']}]} [ocp_resources Role] 2026-07-09T16:01:35.720463Z [info ] Wait until Role http-onnx-view is deleted [ocp_resources Role] 2026-07-09T16:01:35.733678Z [info ] Delete InferenceService http-onnx [ocp_resources InferenceService] 2026-07-09T16:01:35.768468Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'openvino_ir', 'security.opendatahub.io/enable-auth': 'true', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-07-09T15:55:35Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'labels': {'networking.kserve.io/visibility': 'exposed'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-07-09T15:55:35Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:security.opendatahub.io/enable-auth': {}, 'f:serving.kserve.io/deploymentMode': {}}, 'f:labels': {'.': {}, 'f:networking.kserve.io/visibility': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storage': {'.': {}, 'f:key': {}, 'f:path': {}}}, 'f:serviceAccountName': {}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-07-09T16:01:22Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-07-09T16:01:34Z'}], 'name': 'http-onnx', 'namespace': 'test-kserve-raw-token-authentication', 'resourceVersion': '250884299', 'uid': '4eca39bf-2cbc-4507-ad1a-045e0f4b3549'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'openvino_ir'}, 'name': '', 'resources': {}, 'runtime': 'http-mnist-runtime', 'storage': {'key': 'ci-bucket-secret', 'path': '/test-dir/'}}, 'serviceAccountName': 'models-bucket-sa'}}, 'status': {'address': {'url': 'https://http-onnx-predictor.test-kserve-raw-token-authentication.svc.cluster.local:8443'}, 'components': {'predictor': {'url': 'http://http-onnx-predictor-test-kserve-raw-token-authentication.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-07-09T15:55:35Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-07-09T16:01:34Z', 'message': 'ReplicaSet "http-onnx-predictor-74ff48987f" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-07-09T16:01:34Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T15:55:35Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 2}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 3, 'servingRuntimeName': 'http-mnist-runtime', 'url': 'https://http-onnx-test-kserve-raw-token-authentication.apps.ods-qe-psi-23.osp.rh-ods.com'}} [ocp_resources InferenceService] 2026-07-09T16:01:35.784796Z [info ] Wait until InferenceService http-onnx is deleted [ocp_resources InferenceService] 2026-07-09T16:01:36.813885Z [info ] Delete ServiceAccount models-bucket-sa [ocp_resources ServiceAccount] 2026-07-09T16:01:36.840970Z [info ] Deleting {'kind': 'ServiceAccount', 'apiVersion': 'v1', 'metadata': {'name': 'models-bucket-sa', 'namespace': 'test-kserve-raw-token-authentication', 'uid': 'f26b9988-adf1-43f3-943a-9302b1b2be5b', 'resourceVersion': '250875008', 'creationTimestamp': '2026-07-09T15:55:35Z', 'annotations': {'openshift.io/internal-registry-pull-secret-ref': 'models-bucket-sa-dockercfg-22ffw'}, 'managedFields': [{'manager': 'openshift.io/image-registry-pull-secrets_service-account-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-07-09T15:55:35Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:imagePullSecrets': {}, 'f:metadata': {'f:annotations': {'f:openshift.io/internal-registry-pull-secret-ref': {}}}, 'f:secrets': {'k:{"name":"models-bucket-sa-dockercfg-22ffw"}': {}}}}, {'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T15:55:35Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:secrets': {'.': {}, 'k:{"name":"models-bucket-secret"}': {}}}}]}, 'secrets': [{'name': 'models-bucket-secret'}, {'name': 'models-bucket-sa-dockercfg-22ffw'}], 'imagePullSecrets': [{'name': 'models-bucket-sa-dockercfg-22ffw'}]} [ocp_resources ServiceAccount] 2026-07-09T16:01:36.856384Z [info ] Wait until ServiceAccount models-bucket-sa is deleted [ocp_resources ServiceAccount] 2026-07-09T16:01:36.869624Z [info ] Delete Secret models-bucket-secret [ocp_resources Secret] 2026-07-09T16:01:36.907576Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'models-bucket-secret', 'namespace': 'test-kserve-raw-token-authentication', 'uid': '30a655b9-2ee8-4622-ab29-0745965b348f', 'resourceVersion': '250875005', 'creationTimestamp': '2026-07-09T15:55:35Z', 'labels': {'opendatahub.io/dashboard': 'true', 'opendatahub.io/managed': 'true'}, 'annotations': {'opendatahub.io/connection-type': 's3', 'serving.kserve.io/s3-endpoint': 's3.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-1', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-usehttps': '1', 'serving.kserve.io/s3-verifyssl': '0'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T15:55:35Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:AWS_ACCESS_KEY_ID': {}, 'f:AWS_DEFAULT_REGION': {}, 'f:AWS_S3_BUCKET': {}, 'f:AWS_S3_ENDPOINT': {}, 'f:AWS_SECRET_ACCESS_KEY': {}}, 'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/connection-type': {}, 'f:serving.kserve.io/s3-endpoint': {}, 'f:serving.kserve.io/s3-region': {}, 'f:serving.kserve.io/s3-useanoncredential': {}, 'f:serving.kserve.io/s3-usehttps': {}, 'f:serving.kserve.io/s3-verifyssl': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}, 'f:opendatahub.io/managed': {}}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-09T16:01:36.926235Z [info ] Wait until Secret models-bucket-secret is deleted [ocp_resources Secret] 2026-07-09T16:01:36.939500Z [info ] Delete Secret ci-bucket-secret [ocp_resources Secret] 2026-07-09T16:01:36.973460Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'ci-bucket-secret', 'namespace': 'test-kserve-raw-token-authentication', 'uid': '7e5dae52-eb13-4089-9cf4-5e9efd5e7702', 'resourceVersion': '250875003', 'creationTimestamp': '2026-07-09T15:55:35Z', 'labels': {'opendatahub.io/dashboard': 'true', 'opendatahub.io/managed': 'true'}, 'annotations': {'opendatahub.io/connection-type': 's3', 'serving.kserve.io/s3-endpoint': 's3.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-1', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-usehttps': '1', 'serving.kserve.io/s3-verifyssl': '0'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T15:55:35Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:AWS_ACCESS_KEY_ID': {}, 'f:AWS_DEFAULT_REGION': {}, 'f:AWS_S3_BUCKET': {}, 'f:AWS_S3_ENDPOINT': {}, 'f:AWS_SECRET_ACCESS_KEY': {}}, 'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/connection-type': {}, 'f:serving.kserve.io/s3-endpoint': {}, 'f:serving.kserve.io/s3-region': {}, 'f:serving.kserve.io/s3-useanoncredential': {}, 'f:serving.kserve.io/s3-usehttps': {}, 'f:serving.kserve.io/s3-verifyssl': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}, 'f:opendatahub.io/managed': {}}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-09T16:01:36.991374Z [info ] Wait until Secret ci-bucket-secret is deleted [ocp_resources Secret] 2026-07-09T16:01:37.004575Z [info ] Delete ServingRuntime http-mnist-runtime [ocp_resources ServingRuntime] 2026-07-09T16:01:37.032137Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'creationTimestamp': '2026-07-09T15:55:35Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/recommended-accelerators': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-07-09T15:55:35Z'}], 'name': 'http-mnist-runtime', 'namespace': 'test-kserve-raw-token-authentication', 'resourceVersion': '250875001', 'uid': '5b281df1-c3b1-4881-9c94-a8fdbe013397'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-07-09T16:01:37.047416Z [info ] Wait until ServingRuntime http-mnist-runtime is deleted [ocp_resources ServingRuntime] 2026-07-09T16:01:37.148158Z [info ] Delete Project test-kserve-raw-token-authentication [ocp_resources Project] 2026-07-09T16:01:37.185703Z [info ] Deleting {'kind': 'Project', 'apiVersion': 'project.openshift.io/v1', 'metadata': {'name': 'test-kserve-raw-token-authentication', 'uid': '6b1c1cc3-4c6e-41c8-9e2b-53ae15a0ca4e', 'resourceVersion': '250874975', 'creationTimestamp': '2026-07-09T15:55:34Z', 'labels': {'kubernetes.io/metadata.name': 'test-kserve-raw-token-authentication', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/description': '', 'openshift.io/display-name': '', 'openshift.io/requester': 'system:serviceaccount:kube-system:olminstall-cluster-admin', 'openshift.io/sa.scc.mcs': 's0:c38,c17', 'openshift.io/sa.scc.supplemental-groups': '1001440000/10000', 'openshift.io/sa.scc.uid-range': '1001440000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-07-09T15:55:35Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'openshift-apiserver', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T15:55:34Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/description': {}, 'f:openshift.io/display-name': {}, 'f:openshift.io/requester': {}}, 'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'openshift-controller-manager', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T15:55:34Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:finalizers': {}}}, 'subresource': 'finalize'}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T15:55:35Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Project] 2026-07-09T16:01:37.219402Z [info ] Wait until Project test-kserve-raw-token-authentication is deleted [ocp_resources Project] _ 4 of 5 completed, 1 Pass, 2 Fail, 0 Skip, 0 XPass, 0 XFail, 1 Error, 0 ReRun _ tests/model_serving/model_server/kserve/authentication/test_non_admin_users.py::TestRawUnprivilegedUser::test_non_admin_deploy_raw_and_query_model[unprivileged_model_namespace0-unprivileged_s3_ovms_raw_inference_service0]  - test_non_admin_deploy_raw_and_query_model[unprivileged_model_namespace0-unprivileged_s3_ovms_raw_inference_service0] - -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-07-09T16:01:46.437806Z [info ] Executing class fixture: unprivileged_model_namespace [conftest] 2026-07-09T16:01:46.437957Z [info ] kind: ProjectRequest api version: project.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:46.438097Z [info ] Create ProjectRequest test-non-admin-raw [ocp_resources ProjectRequest] 2026-07-09T16:01:46.438211Z [info ] Posting {'apiVersion': 'project.openshift.io/v1', 'kind': 'ProjectRequest', 'metadata': {'name': 'test-non-admin-raw'}} [ocp_resources ProjectRequest] 2026-07-09T16:01:46.536452Z [info ] kind: Project api version: project.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:46.536762Z [info ] Wait for Project test-non-admin-raw status to be Active [ocp_resources Project] 2026-07-09T16:01:46.553568Z [info ] Status of Project test-non-admin-raw is Active [ocp_resources Project] 2026-07-09T16:01:46.553834Z [info ] Executing class fixture: http_s3_ovms_serving_runtime [conftest] 2026-07-09T16:01:46.554088Z [info ] kind: Template api version: template.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:01:46.587530Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:01:46.587698Z [info ] Create ServingRuntime http-mnist-runtime [ocp_resources ServingRuntime] 2026-07-09T16:01:46.587864Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'labels': {'opendatahub.io/dashboard': 'true'}, 'name': 'http-mnist-runtime', 'namespace': 'test-non-admin-raw'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-07-09T16:01:46.623968Z [info ] Executing class fixture: unprivileged_ci_endpoint_s3_secret [conftest] 2026-07-09T16:01:46.639615Z [info ] Create Secret ci-bucket-unprivileged [ocp_resources Secret] 2026-07-09T16:01:46.639797Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'ci-bucket-unprivileged', 'labels': {'opendatahub.io/managed': 'true', 'opendatahub.io/dashboard': 'true'}, 'annotations': {'opendatahub.io/connection-type': 's3', 'serving.kserve.io/s3-endpoint': 's3.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-1', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-verifyssl': '0', 'serving.kserve.io/s3-usehttps': '1'}, 'namespace': 'test-non-admin-raw'}, 'data': '*******'} [ocp_resources Secret] 2026-07-09T16:01:46.686415Z [info ] Wait until Secret ci-bucket-unprivileged is created [ocp_resources Secret] 2026-07-09T16:01:46.705447Z [info ] Executing class fixture: unprivileged_s3_ovms_raw_inference_service [conftest] 2026-07-09T16:01:46.722441Z [info ] kind: InferenceService api version: serving.kserve.io/v1beta1 [ocp_resources.resource] 2026-07-09T16:01:46.722606Z [info ] Create InferenceService http-onnx-raw [ocp_resources InferenceService] 2026-07-09T16:01:46.722765Z [info ] Posting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'name': 'http-onnx-raw', 'annotations': {'serving.kserve.io/deploymentMode': 'RawDeployment'}, 'namespace': 'test-non-admin-raw'}, 'spec': {'predictor': {'model': {'modelFormat': {'name': 'openvino_ir'}, 'version': None, 'runtime': 'http-mnist-runtime', 'storage': {'key': 'ci-bucket-unprivileged', 'path': '/test-dir/'}}}}} [ocp_resources InferenceService] 2026-07-09T16:01:46.793708Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-07-09T16:01:47.820273Z [info ] Waiting for pods to be created [utilities.infra] name=utilities.infra 2026-07-09T16:01:47.872164Z [info ] Verifying no failed pods  [utilities.infra] name=utilities.infra 2026-07-09T16:02:08.149187Z [info ] Waiting for inference deployment replicas to complete [utilities.infra] name=utilities.infra 2026-07-09T16:02:08.204778Z [info ] Wait for Deployment http-onnx-raw-predictor to be deployed: True [ocp_resources Deployment] 2026-07-09T16:02:08.230435Z [info ] Wait for InferenceService/http-onnx-raw's 'Ready' condition to be 'True' [ocp_resources InferenceService] 2026-07-09T16:02:08.230742Z [info ] Wait until InferenceService http-onnx-raw is created [ocp_resources InferenceService] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-07-09T16:02:08.319545Z [info ] kind: ServingRuntime api version: serving.kserve.io/v1alpha1 [ocp_resources.resource] 2026-07-09T16:02:08.345904Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-09T16:02:08.354344Z [info ] kind: Infrastructure api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:02:08.433535Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-07-09T16:02:08.701542Z [info ] Using router certificate secret: router-certs-default [utilities.certificates_utils] name=utilities.certificates_utils [2026-07-09T16:02:08Z WARN kube_client::client::tls::rustls_tls] Server cert bypassed 2026-07-09T16:02:09.052342Z [info ] Inference output: HTTP/1.1 404 Not Found content-length: 53 content-type: application/json; charset=utf-8 access-control-allow-headers: * access-control-allow-methods: * access-control-allow-origin: * {"error":"Model with requested version is not found"} [utilities.inference_utils] name=utilities.inference_utils FAILED TEST: TestRawUnprivilegedUser.test_non_admin_deploy_raw_and_query_model[unprivileged_model_namespace0-unprivileged_s3_ovms_raw_inference_service0] STATUS: FAILED 2026-07-09T16:02:09.057844Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_non_admin_users.py", line 37, in test_non_admin_deploy_raw_and_query_model verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=unprivileged_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... use_default_query=True, ^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx-raw","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} [conftest] ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-07-09T16:02:09.058120Z [info ] Delete InferenceService http-onnx-raw [ocp_resources InferenceService] 2026-07-09T16:02:09.088543Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1beta1', 'kind': 'InferenceService', 'metadata': {'annotations': {'modelFormat': 'openvino_ir', 'serving.kserve.io/deploymentMode': 'Standard'}, 'creationTimestamp': '2026-07-09T16:01:46Z', 'finalizers': ['inferenceservice.finalizers', 'odh.inferenceservice.finalizers'], 'generation': 1, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:serving.kserve.io/deploymentMode': {}}}, 'f:spec': {'.': {}, 'f:predictor': {'.': {}, 'f:model': {'.': {}, 'f:modelFormat': {'.': {}, 'f:name': {}}, 'f:runtime': {}, 'f:storage': {'.': {}, 'f:key': {}, 'f:path': {}}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-07-09T16:01:46Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"inferenceservice.finalizers"': {}, 'v:"odh.inferenceservice.finalizers"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-07-09T16:01:46Z'}, {'apiVersion': 'serving.kserve.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:address': {'.': {}, 'f:url': {}}, 'f:components': {'.': {}, 'f:predictor': {'.': {}, 'f:url': {}}}, 'f:conditions': {}, 'f:deploymentMode': {}, 'f:modelStatus': {'.': {}, 'f:copies': {'.': {}, 'f:failedCopies': {}, 'f:totalCopies': {}}, 'f:states': {'.': {}, 'f:activeModelState': {}, 'f:targetModelState': {}}, 'f:transitionStatus': {}}, 'f:observedGeneration': {}, 'f:servingRuntimeName': {}, 'f:url': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-07-09T16:01:58Z'}], 'name': 'http-onnx-raw', 'namespace': 'test-non-admin-raw', 'resourceVersion': '250885226', 'uid': '2223c222-ef25-4f30-9d99-f4ae7aab57b5'}, 'spec': {'predictor': {'automountServiceAccountToken': False, 'model': {'modelFormat': {'name': 'openvino_ir'}, 'name': '', 'resources': {}, 'runtime': 'http-mnist-runtime', 'storage': {'key': 'ci-bucket-unprivileged', 'path': '/test-dir/'}}}}, 'status': {'address': {'url': 'http://http-onnx-raw-predictor.test-non-admin-raw.svc.cluster.local:8080'}, 'components': {'predictor': {'url': 'http://http-onnx-raw-predictor-test-non-admin-raw.example.com'}}, 'conditions': [{'lastTransitionTime': '2026-07-09T16:01:46Z', 'status': 'True', 'type': 'IngressReady'}, {'lastTransitionTime': '2026-07-09T16:01:58Z', 'message': 'ReplicaSet "http-onnx-raw-predictor-59f5884774" has successfully progressed.', 'reason': 'NewReplicaSetAvailable', 'status': 'True', 'type': 'PredictorReady'}, {'lastTransitionTime': '2026-07-09T16:01:58Z', 'status': 'True', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T16:01:46Z', 'severity': 'Info', 'status': 'False', 'type': 'Stopped'}], 'deploymentMode': 'Standard', 'modelStatus': {'copies': {'failedCopies': 0, 'totalCopies': 1}, 'states': {'activeModelState': 'Loaded', 'targetModelState': 'Loaded'}, 'transitionStatus': 'UpToDate'}, 'observedGeneration': 1, 'servingRuntimeName': 'http-mnist-runtime', 'url': 'http://http-onnx-raw-predictor.test-non-admin-raw.svc.cluster.local'}} [ocp_resources InferenceService] 2026-07-09T16:02:09.108453Z [info ] Wait until InferenceService http-onnx-raw is deleted [ocp_resources InferenceService] 2026-07-09T16:02:10.139089Z [info ] Delete Secret ci-bucket-unprivileged [ocp_resources Secret] 2026-07-09T16:02:10.178013Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'ci-bucket-unprivileged', 'namespace': 'test-non-admin-raw', 'uid': '30f54125-b73b-4395-bec4-674b278d9d13', 'resourceVersion': '250884851', 'creationTimestamp': '2026-07-09T16:01:46Z', 'labels': {'opendatahub.io/dashboard': 'true', 'opendatahub.io/managed': 'true'}, 'annotations': {'opendatahub.io/connection-type': 's3', 'serving.kserve.io/s3-endpoint': 's3.amazonaws.com/', 'serving.kserve.io/s3-region': 'us-east-1', 'serving.kserve.io/s3-useanoncredential': 'false', 'serving.kserve.io/s3-usehttps': '1', 'serving.kserve.io/s3-verifyssl': '0'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:01:46Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:AWS_ACCESS_KEY_ID': {}, 'f:AWS_DEFAULT_REGION': {}, 'f:AWS_S3_BUCKET': {}, 'f:AWS_S3_ENDPOINT': {}, 'f:AWS_SECRET_ACCESS_KEY': {}}, 'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/connection-type': {}, 'f:serving.kserve.io/s3-endpoint': {}, 'f:serving.kserve.io/s3-region': {}, 'f:serving.kserve.io/s3-useanoncredential': {}, 'f:serving.kserve.io/s3-usehttps': {}, 'f:serving.kserve.io/s3-verifyssl': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}, 'f:opendatahub.io/managed': {}}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-09T16:02:10.193488Z [info ] Wait until Secret ci-bucket-unprivileged is deleted [ocp_resources Secret] 2026-07-09T16:02:10.217648Z [info ] Delete ServingRuntime http-mnist-runtime [ocp_resources ServingRuntime] 2026-07-09T16:02:10.247689Z [info ] Deleting {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'ServingRuntime', 'metadata': {'annotations': {'opendatahub.io/recommended-accelerators': '["nvidia.com/gpu"]', 'opendatahub.io/runtime-version': 'v2026.1.0', 'openshift.io/display-name': 'OpenVINO Model Server'}, 'creationTimestamp': '2026-07-09T16:01:46Z', 'generation': 1, 'labels': {'opendatahub.io/dashboard': 'true'}, 'managedFields': [{'apiVersion': 'serving.kserve.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:opendatahub.io/recommended-accelerators': {}, 'f:opendatahub.io/runtime-version': {}, 'f:openshift.io/display-name': {}}, 'f:labels': {'.': {}, 'f:opendatahub.io/dashboard': {}}}, 'f:spec': {'.': {}, 'f:annotations': {'.': {}, 'f:opendatahub.io/kserve-runtime': {}, 'f:prometheus.io/path': {}, 'f:prometheus.io/port': {}}, 'f:containers': {}, 'f:multiModel': {}, 'f:protocolVersions': {}, 'f:supportedModelFormats': {}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-07-09T16:01:46Z'}], 'name': 'http-mnist-runtime', 'namespace': 'test-non-admin-raw', 'resourceVersion': '250884849', 'uid': '49b7e0e7-c34d-497e-8b79-b6d518c15838'}, 'spec': {'annotations': {'opendatahub.io/kserve-runtime': 'ovms', 'prometheus.io/path': '/metrics', 'prometheus.io/port': '8888'}, 'containers': [{'args': ['--model_name={{.Name}}', '--port=8001', '--rest_port=8888', '--model_path=/mnt/models', '--file_system_poll_wait_seconds=0', '--metrics_enable'], 'image': 'registry.redhat.io/rhoai/odh-openvino-model-server-rhel9@sha256:8c1b225509bfdc43756195f34a356d1e2ebd58fcd83fda3d37380dd7f5304df8', 'name': 'kserve-container', 'ports': [{'containerPort': 8888, 'protocol': 'TCP'}]}], 'multiModel': False, 'protocolVersions': ['v2', 'grpc-v2'], 'supportedModelFormats': [{'autoSelect': True, 'name': 'openvino_ir', 'version': 'opset13'}, {'name': 'onnx', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '1'}, {'autoSelect': True, 'name': 'tensorflow', 'version': '2'}, {'autoSelect': True, 'name': 'paddle', 'version': '2'}, {'autoSelect': True, 'name': 'pytorch', 'version': '2'}]}} [ocp_resources ServingRuntime] 2026-07-09T16:02:10.266012Z [info ] Wait until ServingRuntime http-mnist-runtime is deleted [ocp_resources ServingRuntime] 2026-07-09T16:02:10.330630Z [info ] Delete Project test-non-admin-raw [ocp_resources Project] 2026-07-09T16:02:10.389267Z [info ] Deleting {'kind': 'Project', 'apiVersion': 'project.openshift.io/v1', 'metadata': {'name': 'test-non-admin-raw', 'uid': 'c0e00a36-bf99-4909-a84e-330f54db01e4', 'resourceVersion': '250884838', 'creationTimestamp': '2026-07-09T16:01:46Z', 'labels': {'kubernetes.io/metadata.name': 'test-non-admin-raw', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/description': '', 'openshift.io/display-name': '', 'openshift.io/requester': 'system:serviceaccount:kube-system:olminstall-cluster-admin', 'openshift.io/sa.scc.mcs': 's0:c38,c22', 'openshift.io/sa.scc.supplemental-groups': '1001450000/10000', 'openshift.io/sa.scc.uid-range': '1001450000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-07-09T16:01:46Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:01:46Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}, {'manager': 'openshift-apiserver', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:01:46Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/description': {}, 'f:openshift.io/display-name': {}, 'f:openshift.io/requester': {}}, 'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'openshift-controller-manager', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-09T16:01:46Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:finalizers': {}}}, 'subresource': 'finalize'}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Project] 2026-07-09T16:02:10.656822Z [info ] Wait until Project test-non-admin-raw is deleted [ocp_resources Project] _ TestRawUnprivilegedUser.test_non_admin_deploy_raw_and_query_model[unprivileged_model_namespace0-unprivileged_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_non_admin_users.py", line 37, in test_non_admin_deploy_raw_and_query_model verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=unprivileged_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... use_default_query=True, ^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx-raw","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} _ 5 of 5 completed, 1 Pass, 3 Fail, 0 Skip, 0 XPass, 0 XFail, 1 Error, 0 ReRun _ = 3 failed, 1 passed, 219 deselected, 725 warnings, 1 error in 409.96s (0:06:49) = ==================================== ERRORS ==================================== _ ERROR at setup of TestKserveTokenAuthenticationRawForRest.test_disabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 170, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 536, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1710, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 806, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 630, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 726, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1232, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 924, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1328, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 998, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/conftest.py", line 100, in patched_remove_raw_authentication_isvc wait_for_raw_isvc_https_infer_ready(isvc=http_s3_ovms_raw_inference_service, token=None) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 234, in wait_for_raw_isvc_https_infer_ready for ok in TimeoutSampler(wait_timeout=timeout, sleep=sleep, func=_https_infer_ok): ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 300 Function: tests.model_serving.model_server.utils._https_infer_ok Last exception: N/A. =================================== FAILURES =================================== _ TestKserveTokenAuthenticationRawForRest.test_model_authentication_using_rest_raw[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py", line 39, in test_model_authentication_using_rest_raw verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=http_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... token=http_raw_inference_token, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} _ TestKserveTokenAuthenticationRawForRest.test_re_enabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py", line 61, in test_re_enabled_raw_model_authentication verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=http_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... token=http_raw_inference_token, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} _ TestRawUnprivilegedUser.test_non_admin_deploy_raw_and_query_model[unprivileged_model_namespace0-unprivileged_s3_ovms_raw_inference_service0] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/model_serving/model_server/kserve/authentication/test_non_admin_users.py", line 37, in test_non_admin_deploy_raw_and_query_model verify_inference_response( ~~~~~~~~~~~~~~~~~~~~~~~~~^ inference_service=unprivileged_s3_ovms_raw_inference_service, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... use_default_query=True, ^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/model_serving/model_server/utils.py", line 154, in verify_inference_response assert re.search(expected_response_text, formatted_res), ( # type: ignore[arg-type] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Expected: {"model_name":"http-onnx-raw","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} =============================== warnings summary =============================== tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py: 642 warnings tests/model_serving/model_server/kserve/authentication/test_non_admin_users.py: 83 warnings /workspace/tests-shared/tests-payload/.tools/python/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: /workspace/tests-shared/tests-payload/results/model_server-smoke.xml - =========================== short test summary info ============================ ERROR tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_disabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] - timeout_sampler.TimeoutExpiredError: Timed Out: 300 Function: tests.model_serving.model_server.utils._https_infer_ok Last exception: N/A. FAILED tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_model_authentication_using_rest_raw[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] - AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} FAILED tests/model_serving/model_server/kserve/authentication/test_kserve_token_authentication_raw.py::TestKserveTokenAuthenticationRawForRest::test_re_enabled_raw_model_authentication[unprivileged_model_namespace0-http_s3_ovms_raw_inference_service0] - AssertionError: Expected: {"model_name":"http-onnx","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} FAILED tests/model_serving/model_server/kserve/authentication/test_non_admin_users.py::TestRawUnprivilegedUser::test_non_admin_deploy_raw_and_query_model[unprivileged_model_namespace0-unprivileged_s3_ovms_raw_inference_service0] - AssertionError: Expected: {"model_name":"http-onnx-raw","model_version":"1","outputs":\[\{"name":"Plus214_Output_0","shape":\[1,10\],"datatype":"FP32","data":\[.*\]}]} not found in: {"error":"Modelwithrequestedversionisnotfound"} = 3 failed, 1 passed, 219 deselected, 725 warnings, 1 error in 409.97s (0:06:49) =