--- BVT pytest attempt 2026-07-05T07:26:31.567767+00:00 (/.local/bin/uv --directory /home/odh/opendatahub-tests...) --- 2026-07-05T07:26:33.685286Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-07-05T07:26:33.685827Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-07-05T07:26:33.778194Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-07-05T07:26:33.825784Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( ============================= test session starts ============================== platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python3 cachedir: .pytest_cache metadata: {'Python': '3.14.5', 'Platform': 'Linux-5.14.0-570.120.1.el9_6.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.0.3', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.13.0', 'langsmith': '0.8.5', 'asyncio': '1.3.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.4.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.2.0'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.13.0, langsmith-0.8.5, asyncio-1.3.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.4.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.2.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... collected 35 items / 31 deselected / 4 selected tests/ogx/operator/test_ogx_server.py::TestOgxServer::test_ogx_server_verify_images[unprivileged_model_namespace0-ogx_server0]  ----------------------- test_ogx_server_verify_images[unprivileged_model_namespace0-ogx_server0] ----------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-07-05T07:26:35.303410Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-07-05T07:26:35.303601Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-07-05T07:26:35.305027Z [info ] Executing session fixture: admin_client [conftest] 2026-07-05T07:26:35.305200Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-07-05T07:26:35.305358Z [info ] Executing session fixture: dsc_resource [conftest] 2026-07-05T07:26:35.305497Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-07-05T07:26:35.360857Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-07-05T07:26:35.361001Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-07-05T07:26:35.361862Z [info ] Executing session fixture: os_path_environment [conftest] 2026-07-05T07:26:35.362000Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-07-05T07:26:35.362128Z [info ] Executing session fixture: bin_directory [conftest] 2026-07-05T07:26:35.363091Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-07-05T07:26:35.374426Z [info ] kind: ConsoleCLIDownload api version: console.openshift.io/v1 [ocp_resources.resource] 2026-07-05T07:26:35.395035Z [info ] All oc console cli download links: ['https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar'] [utilities.infra] name=utilities.infra 2026-07-05T07:26:35.395113Z [info ] Downloading archive using: url=https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar [utilities.infra] name=utilities.infra 2026-07-05T07:26:44.768821Z [info ] Extract the downloaded archive. [utilities.infra] name=utilities.infra 2026-07-05T07:26:44.955564Z [info ] Downloaded file: ['oc']  [utilities.infra] name=utilities.infra 2026-07-05T07:26:44.983199Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-07-05T07:26:44.981260Z [info ] OC binary path: /tmp/pytest-gFeu5bW3GmHQoduSszCCup/bin0/oc [tests.conftest] name=tests.conftest 2026-07-05T07:26:44.981350Z [info ] Adding /tmp/pytest-gFeu5bW3GmHQoduSszCCup/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-07-05T07:26:44.983633Z [info ] Executing session fixture: nodes [conftest] 2026-07-05T07:26:45.748351Z [info ] Executing session fixture: dsci_resource [conftest] 2026-07-05T07:26:45.748532Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-07-05T07:26:45.759942Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-07-05T07:26:45.760105Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-07-05T07:26:45.760254Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-07-05T07:26:45.759362Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-07-05T07:26:45.760553Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-07-05T07:26:45.834512Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-07-05T07:26:45.907006Z [warning ] Skipping RHOAI resource checks, got --cluster-sanity-skip-rhoai-check [utilities.infra] name=utilities.infra 2026-07-05T07:26:45.907975Z [info ] Executing session fixture: autouse_fixtures [conftest] 2026-07-05T07:26:45.908130Z [info ] Executing session fixture: openshift_version [conftest] 2026-07-05T07:26:45.938160Z [info ] kind: ClusterVersion api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-05T07:26:45.974279Z [info ] Executing session fixture: skip_ogx_if_not_supported_openshift_version [conftest] 2026-07-05T07:26:45.974443Z [info ] Executing session fixture: related_images_refs [conftest] 2026-07-05T07:26:46.013255Z [info ] kind: ClusterServiceVersion api version: operators.coreos.com/v1alpha1 [ocp_resources.resource] 2026-07-05T07:26:46.074436Z [info ] Looking for rhods-operator.3.5.0-ea.1 CSV in namespace redhat-ods-applications [utilities.operator_utils] name=utilities.operator_utils 2026-07-05T07:26:46.138076Z [info ] Found cluster service version: rhods-operator.3.5.0-ea.1 [utilities.operator_utils] name=utilities.operator_utils 2026-07-05T07:26:46.162114Z [info ] Executing session fixture: pytestconfig [conftest] 2026-07-05T07:26:46.162280Z [info ] Executing session fixture: use_unprivileged_client [conftest] 2026-07-05T07:26:46.162440Z [info ] Executing session fixture: kubconfig_filepath [conftest] 2026-07-05T07:26:46.162594Z [info ] Executing session fixture: cluster_authentication [conftest] 2026-07-05T07:26:46.162847Z [info ] kind: Authentication api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-05T07:26:46.174072Z [info ] Executing session fixture: is_byoidc [conftest] 2026-07-05T07:26:46.187854Z [info ] Executing session fixture: non_admin_user_password [conftest] 2026-07-05T07:26:46.188028Z [info ] Executing session fixture: unprivileged_client [conftest] 2026-07-05T07:26:46.185887Z [warning ] Unprivileged client is not enabled, using admin client [tests.conftest] name=tests.conftest 2026-07-05T07:26:46.188329Z [info ] Executing session fixture: teardown_resources [conftest] 2026-07-05T07:26:46.188480Z [info ] Executing class fixture: distribution_name [conftest] 2026-07-05T07:26:46.188627Z [info ] Executing class fixture: unprivileged_model_namespace [conftest] 2026-07-05T07:26:46.188781Z [info ] kind: ProjectRequest api version: project.openshift.io/v1 [ocp_resources.resource] 2026-07-05T07:26:46.188901Z [info ] Create ProjectRequest test-ogx-operator-09c9 [ocp_resources ProjectRequest] 2026-07-05T07:26:46.189016Z [info ] Posting {'apiVersion': 'project.openshift.io/v1', 'kind': 'ProjectRequest', 'metadata': {'name': 'test-ogx-operator-09c9'}} [ocp_resources ProjectRequest] 2026-07-05T07:26:46.286156Z [info ] kind: Project api version: project.openshift.io/v1 [ocp_resources.resource] 2026-07-05T07:26:46.286311Z [info ] Wait for Project test-ogx-operator-09c9 status to be Active [ocp_resources Project] 2026-07-05T07:26:46.286443Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-07-05T07:26:46.303355Z [info ] Status of Project test-ogx-operator-09c9 is Active [ocp_resources Project] 2026-07-05T07:26:46.303504Z [info ] Elapsed time: 5.1021575927734375e-05 [0:00:00.000051] [timeout_sampler] 2026-07-05T07:26:46.303645Z [info ] Executing class fixture: enabled_ogx_operator [conftest] 2026-07-05T07:26:46.319033Z [warning ] Component ogx was already set to managementState Managed [utilities.data_science_cluster_utils] name=utilities.data_science_cluster_utils 2026-07-05T07:26:46.320124Z [info ] Executing class fixture: vector_io_provider_deployment_config_factory [conftest] 2026-07-05T07:26:46.320266Z [info ] Executing class fixture: files_provider_config_factory [conftest] 2026-07-05T07:26:46.320408Z [info ] Executing class fixture: is_disconnected_cluster [conftest] 2026-07-05T07:26:46.320549Z [info ] kind: OperatorHub api version: config.openshift.io/v1 [ocp_resources.resource] 2026-07-05T07:26:46.342402Z [info ] Disconnected cluster detection: False [utilities.infra] name=utilities.infra 2026-07-05T07:26:46.345407Z [info ] Executing class fixture: ogx_server_secret [conftest] 2026-07-05T07:26:46.345558Z [info ] Create Secret ogx-distribution-secret [ocp_resources Secret] 2026-07-05T07:26:46.345731Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'ogx-distribution-secret', 'namespace': 'test-ogx-operator-09c9'}, 'stringData': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-05T07:26:46.371727Z [info ] Executing class fixture: postgres_deployment [conftest] 2026-07-05T07:26:46.371899Z [info ] kind: Deployment api version: apps/v1 [ocp_resources.resource] 2026-07-05T07:26:46.372048Z [info ] Create Deployment vector-io-postgres-deployment [ocp_resources Deployment] 2026-07-05T07:26:46.372213Z [info ] Posting {'apiVersion': 'apps/v1', 'kind': 'Deployment', 'metadata': {'name': 'vector-io-postgres-deployment', 'namespace': 'test-ogx-operator-09c9'}, 'spec': {'selector': {'matchLabels': {'app': 'postgres'}}, 'template': {'metadata': {'labels': {'app': 'postgres'}}, 'spec': {'containers': [{'name': 'postgres', 'image': 'registry.redhat.io/rhel9/postgresql-15@sha256:90ec347a35ab8a5d530c8d09f5347b13cc71df04f3b994bfa8b1a409b1171d59', 'ports': [{'containerPort': 5432}], 'env': [{'name': 'POSTGRESQL_DATABASE', 'value': 'ps_db'}, {'name': 'POSTGRESQL_USER', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'postgres-user'}}}, {'name': 'POSTGRESQL_PASSWORD', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'postgres-password'}}}], 'volumeMounts': [{'name': 'postgresdata', 'mountPath': '/var/lib/pgsql/data'}]}], 'volumes': [{'name': 'postgresdata', 'emptyDir': {}}]}}, 'minReadySeconds': 5, 'replicas': 1, 'strategy': {'type': 'Recreate'}}} [ocp_resources Deployment] 2026-07-05T07:26:46.410374Z [info ] Wait for Deployment vector-io-postgres-deployment to be deployed: True [ocp_resources Deployment] 2026-07-05T07:26:46.410527Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-07-05T07:26:46.425177Z [info ] Elapsed time: 3.743171691894531e-05 [0:00:00.000037] [timeout_sampler] 2026-07-05T07:26:46.425335Z [info ] Waiting for 239.9851779937744 seconds [0:03:59.985178], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-07-05T07:26:52.517145Z [info ] Elapsed time: 6.0786120891571045 [0:00:06.078612] [timeout_sampler] 2026-07-05T07:26:52.517348Z [info ] Executing class fixture: postgres_service [conftest] 2026-07-05T07:26:52.517508Z [info ] Create Service vector-io-postgres-service [ocp_resources Service] 2026-07-05T07:26:52.517659Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'vector-io-postgres-service', 'namespace': 'test-ogx-operator-09c9'}, 'spec': {'ports': [{'port': 5432, 'targetPort': 5432}], 'selector': {'app': 'postgres'}}} [ocp_resources Service] 2026-07-05T07:26:52.552822Z [info ] Wait until Service vector-io-postgres-service is created [ocp_resources Service] 2026-07-05T07:26:52.552977Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-07-05T07:26:52.566273Z [info ] Elapsed time: 3.600120544433594e-05 [0:00:00.000036] [timeout_sampler] 2026-07-05T07:26:52.779827Z [info ] Executing class fixture: ogx_server [conftest] 2026-07-05T07:26:52.780013Z [info ] Executing class fixture: etcd_deployment [conftest] 2026-07-05T07:26:52.780141Z [info ] Create Deployment vector-io-etcd-deployment [ocp_resources Deployment] 2026-07-05T07:26:52.780280Z [info ] Posting {'apiVersion': 'apps/v1', 'kind': 'Deployment', 'metadata': {'name': 'vector-io-etcd-deployment', 'namespace': 'test-ogx-operator-09c9'}, 'spec': {'selector': {'matchLabels': {'app': 'etcd'}}, 'template': {'metadata': {'labels': {'app': 'etcd'}}, 'spec': {'containers': [{'name': 'etcd', 'image': 'quay.io/coreos/etcd@sha256:3397341272b9e0a6f44d7e3fc7c321c6efe6cbe82ce866b9b01d0c704bfc5bf3', 'command': ['etcd', '--advertise-client-urls=http://vector-io-etcd-service:2379', '--listen-client-urls=http://0.0.0.0:2379', '--data-dir=/etcd'], 'ports': [{'containerPort': 2379}], 'volumeMounts': [{'name': 'etcd-data', 'mountPath': '/etcd'}], 'env': [{'name': 'ETCD_AUTO_COMPACTION_MODE', 'value': 'revision'}, {'name': 'ETCD_AUTO_COMPACTION_RETENTION', 'value': '1000'}, {'name': 'ETCD_QUOTA_BACKEND_BYTES', 'value': '4294967296'}, {'name': 'ETCD_SNAPSHOT_COUNT', 'value': '50000'}]}], 'volumes': [{'name': 'etcd-data', 'emptyDir': {}}]}}, 'replicas': 1, 'strategy': {'type': 'Recreate'}}} [ocp_resources Deployment] 2026-07-05T07:26:52.821723Z [info ] Wait for Deployment vector-io-etcd-deployment to be deployed: True [ocp_resources Deployment] 2026-07-05T07:26:52.821942Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-07-05T07:26:52.834872Z [info ] Elapsed time: 4.1484832763671875e-05 [0:00:00.000041] [timeout_sampler] 2026-07-05T07:26:52.835033Z [info ] Waiting for 119.98682141304016 seconds [0:01:59.986821], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-07-05T07:26:56.903260Z [info ] Elapsed time: 4.0474512577056885 [0:00:04.047451] [timeout_sampler] 2026-07-05T07:26:56.903453Z [info ] Executing class fixture: etcd_service [conftest] 2026-07-05T07:26:56.903589Z [info ] Create Service vector-io-etcd-service [ocp_resources Service] 2026-07-05T07:26:56.903756Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'vector-io-etcd-service', 'namespace': 'test-ogx-operator-09c9'}, 'spec': {'ports': [{'port': 2379, 'targetPort': 2379}], 'selector': {'app': 'etcd'}}} [ocp_resources Service] 2026-07-05T07:26:56.965082Z [info ] Wait until Service vector-io-etcd-service is created [ocp_resources Service] 2026-07-05T07:26:56.965249Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-07-05T07:26:56.978471Z [info ] Elapsed time: 4.6253204345703125e-05 [0:00:00.000046] [timeout_sampler] 2026-07-05T07:26:56.978631Z [info ] Executing class fixture: vector_io_secret [conftest] 2026-07-05T07:26:56.978778Z [info ] Create Secret vector-io-secret [ocp_resources Secret] 2026-07-05T07:26:56.978901Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'vector-io-secret', 'namespace': 'test-ogx-operator-09c9'}, 'stringData': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-05T07:26:57.012957Z [info ] Executing class fixture: remote_milvus_deployment [conftest] 2026-07-05T07:26:57.013260Z [info ] Create Deployment vector-io-milvus-deployment [ocp_resources Deployment] 2026-07-05T07:26:57.013564Z [info ] Posting {'apiVersion': 'apps/v1', 'kind': 'Deployment', 'metadata': {'name': 'vector-io-milvus-deployment', 'namespace': 'test-ogx-operator-09c9'}, 'spec': {'selector': {'matchLabels': {'app': 'milvus-standalone'}}, 'template': {'metadata': {'labels': {'app': 'milvus-standalone'}}, 'spec': {'containers': [{'name': 'milvus-standalone', 'image': 'quay.io/jgarciao/milvus@sha256:b7ae4fa7475f810837c91b61eb5aa39fd89f22348823fc76d9ce776a6ddc5844', 'args': ['milvus', 'run', 'standalone'], 'ports': [{'containerPort': 19530, 'protocol': 'TCP'}], 'volumeMounts': [{'name': 'milvus-data', 'mountPath': '/var/lib/milvus'}], 'env': [{'name': 'DEPLOY_MODE', 'value': 'standalone'}, {'name': 'ETCD_ENDPOINTS', 'value': 'vector-io-etcd-service:2379'}, {'name': 'MINIO_ADDRESS', 'value': ''}, {'name': 'COMMON_STORAGETYPE', 'value': 'local'}]}], 'volumes': [{'name': 'milvus-data', 'emptyDir': {}}]}}, 'minReadySeconds': 5, 'replicas': 1, 'strategy': {'type': 'Recreate'}}} [ocp_resources Deployment] 2026-07-05T07:26:57.063786Z [info ] Wait for Deployment vector-io-milvus-deployment to be deployed: True [ocp_resources Deployment] 2026-07-05T07:26:57.064172Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-07-05T07:26:57.089679Z [info ] Elapsed time: 3.814697265625e-05 [0:00:00.000038] [timeout_sampler] 2026-07-05T07:26:57.089850Z [info ] Waiting for 239.97450947761536 seconds [0:03:59.974509], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-07-05T07:27:29.502259Z [info ] Elapsed time: 32.399811029434204 [0:00:32.399811] [timeout_sampler] 2026-07-05T07:27:29.502805Z [info ] Executing class fixture: milvus_service [conftest] 2026-07-05T07:27:29.502994Z [info ] Create Service vector-io-milvus-service [ocp_resources Service] 2026-07-05T07:27:29.503120Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'vector-io-milvus-service', 'namespace': 'test-ogx-operator-09c9'}, 'spec': {'ports': [{'name': 'grpc', 'port': 19530, 'targetPort': 19530}], 'selector': {'app': 'milvus-standalone'}}} [ocp_resources Service] 2026-07-05T07:27:29.534681Z [info ] Wait until Service vector-io-milvus-service is created [ocp_resources Service] 2026-07-05T07:27:29.534971Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-07-05T07:27:29.546286Z [info ] Elapsed time: 3.62396240234375e-05 [0:00:00.000036] [timeout_sampler] 2026-07-05T07:27:29.571622Z [info ] kind: OGXServer api version: ogx.io/v1beta1 --- [DuplicateFilter: Last log `kind: Deployment api version: apps/v1` repeated 2 times] [ocp_resources.resource] 2026-07-05T07:27:29.571985Z [info ] Create OGXServer ogx-server-500ac9b4 [ocp_resources OGXServer] 2026-07-05T07:27:29.572148Z [info ] Posting {'apiVersion': 'ogx.io/v1beta1', 'kind': 'OGXServer', 'metadata': {'name': 'ogx-server-500ac9b4', 'namespace': 'test-ogx-operator-09c9'}, 'spec': {'distribution': {'name': 'rh-dev'}, 'workload': {'resources': {'requests': {'cpu': '1', 'memory': '1Gi'}, 'limits': {'cpu': '2', 'memory': '2Gi'}}, 'overrides': {'env': [{'name': 'INFERENCE_MODEL', 'value': 'Qwen3.6-35B-A3B'}, {'name': 'VLLM_API_TOKEN', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'vllm-api-token'}}}, {'name': 'VLLM_URL', 'value': 'https://litellm-litemaas.apps.prod.rhoai.rh-aiservices-bu.com/v1'}, {'name': 'VLLM_TLS_VERIFY', 'value': 'false'}, {'name': 'VLLM_MAX_TOKENS', 'value': '80000'}, {'name': 'EMBEDDING_MODEL', 'value': 'Nomic-embed-text-v2-moe'}, {'name': 'EMBEDDING_PROVIDER_MODEL_ID', 'value': 'Nomic-embed-text-v2-moe'}, {'name': 'VLLM_EMBEDDING_URL', 'value': 'https://litellm-litemaas.apps.prod.rhoai.rh-aiservices-bu.com/v1'}, {'name': 'VLLM_EMBEDDING_API_TOKEN', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'vllm-embedding-api-token'}}}, {'name': 'VLLM_EMBEDDING_MAX_TOKENS', 'value': '1024'}, {'name': 'VLLM_EMBEDDING_TLS_VERIFY', 'value': 'false'}, {'name': 'POSTGRES_HOST', 'value': 'vector-io-postgres-service'}, {'name': 'POSTGRES_PORT', 'value': '5432'}, {'name': 'POSTGRES_USER', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'postgres-user'}}}, {'name': 'POSTGRES_PASSWORD', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'postgres-password'}}}, {'name': 'POSTGRES_DB', 'value': 'ps_db'}, {'name': 'POSTGRES_TABLE_NAME', 'value': 'ogx_kvstore'}, {'name': 'MILVUS_ENDPOINT', 'value': 'http://vector-io-milvus-service:19530'}, {'name': 'MILVUS_TOKEN', 'valueFrom': {'secretKeyRef': {'name': 'vector-io-secret', 'key': 'milvus-token'}}}, {'name': 'MILVUS_CONSISTENCY_LEVEL', 'value': 'Bounded'}]}, 'storage': {'size': '2Gi'}}, 'network': {'policy': {'ingress': [{'from': [{'namespaceSelector': {'matchLabels': {'kubernetes.io/metadata.name': 'openshift-ingress'}}}], 'ports': [{'protocol': 'TCP', 'port': 8321}]}]}}}} [ocp_resources OGXServer] 2026-07-05T07:27:29.605732Z [info ] Wait until OGXServer ogx-server-500ac9b4 is created [ocp_resources OGXServer] 2026-07-05T07:27:29.605999Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-07-05T07:27:29.617777Z [info ] Elapsed time: 3.790855407714844e-05 [0:00:00.000038] [timeout_sampler] 2026-07-05T07:27:29.618007Z [info ] Wait for OGXServer ogx-server-500ac9b4 status to be Ready [ocp_resources OGXServer] 2026-07-05T07:27:29.618131Z [info ] Waiting for 600 seconds [0:10:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-07-05T07:27:30.641365Z [info ] Status of OGXServer ogx-server-500ac9b4 is Initializing [ocp_resources OGXServer] 2026-07-05T07:28:55.837817Z [info ] Status of OGXServer ogx-server-500ac9b4 is Ready [ocp_resources OGXServer] 2026-07-05T07:28:55.838084Z [info ] Elapsed time: 86.19546890258789 [0:01:26.195469] [timeout_sampler] 2026-07-05T07:28:55.838219Z [info ] Executing class fixture: ogx_server_pods [conftest] 2026-07-05T07:28:55.838343Z [info ] Waiting for 60 seconds [0:01:00], retry every 5 seconds. (Function: utilities.general.wait_for_pods_by_labels Kwargs: {'admin_client': , 'namespace': 'test-ogx-operator-09c9', 'label_selector': 'app=ogx', 'expected_num_pods': 1}) [timeout_sampler] 2026-07-05T07:28:55.854061Z [info ] Elapsed time: 5.6743621826171875e-05 [0:00:00.000057] [timeout_sampler] --------------------------------------------------------- CALL --------------------------------------------------------- PASSED TEST: TestOgxServer.test_ogx_server_verify_images[unprivileged_model_namespace0-ogx_server0] STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-07-05T07:28:55.890136Z [info ] Delete OGXServer ogx-server-500ac9b4 [ocp_resources OGXServer] 2026-07-05T07:28:55.918817Z [info ] Deleting {'apiVersion': 'ogx.io/v1beta1', 'kind': 'OGXServer', 'metadata': {'creationTimestamp': '2026-07-05T07:27:29Z', 'generation': 1, 'managedFields': [{'apiVersion': 'ogx.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'.': {}, 'f:distribution': {'.': {}, 'f:name': {}}, 'f:network': {'.': {}, 'f:policy': {'.': {}, 'f:enabled': {}, 'f:ingress': {}}, 'f:port': {}}, 'f:workload': {'.': {}, 'f:overrides': {'.': {}, 'f:env': {}}, 'f:replicas': {}, 'f:resources': {'.': {}, 'f:limits': {'.': {}, 'f:cpu': {}, 'f:memory': {}}, 'f:requests': {'.': {}, 'f:cpu': {}, 'f:memory': {}}}, 'f:storage': {'.': {}, 'f:mountPath': {}, 'f:size': {}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-07-05T07:27:29Z'}, {'apiVersion': 'ogx.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:availableReplicas': {}, 'f:conditions': {}, 'f:distributionConfig': {'.': {}, 'f:activeDistribution': {}, 'f:availableDistributions': {'.': {}, 'f:rh-dev': {}}}, 'f:phase': {}, 'f:serviceURL': {}, 'f:version': {'.': {}, 'f:lastUpdated': {}, 'f:operatorVersion': {}}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-07-05T07:28:54Z'}], 'name': 'ogx-server-500ac9b4', 'namespace': 'test-ogx-operator-09c9', 'resourceVersion': '241450075', 'uid': '54d54f4c-52f9-42d4-974b-f8f51eaba72f'}, 'spec': {'distribution': {'name': 'rh-dev'}, 'network': {'policy': {'enabled': True, 'ingress': [{'from': [{'namespaceSelector': {'matchLabels': {'kubernetes.io/metadata.name': 'openshift-ingress'}}}], 'ports': [{'port': 8321, 'protocol': 'TCP'}]}]}, 'port': 8321}, 'workload': {'overrides': {'env': [{'name': 'INFERENCE_MODEL', 'value': 'Qwen3.6-35B-A3B'}, {'name': 'VLLM_API_TOKEN', 'valueFrom': {'secretKeyRef': {'key': 'vllm-api-token', 'name': 'ogx-distribution-secret'}}}, {'name': 'VLLM_URL', 'value': 'https://litellm-litemaas.apps.prod.rhoai.rh-aiservices-bu.com/v1'}, {'name': 'VLLM_TLS_VERIFY', 'value': 'false'}, {'name': 'VLLM_MAX_TOKENS', 'value': '80000'}, {'name': 'EMBEDDING_MODEL', 'value': 'Nomic-embed-text-v2-moe'}, {'name': 'EMBEDDING_PROVIDER_MODEL_ID', 'value': 'Nomic-embed-text-v2-moe'}, {'name': 'VLLM_EMBEDDING_URL', 'value': 'https://litellm-litemaas.apps.prod.rhoai.rh-aiservices-bu.com/v1'}, {'name': 'VLLM_EMBEDDING_API_TOKEN', 'valueFrom': {'secretKeyRef': {'key': 'vllm-embedding-api-token', 'name': 'ogx-distribution-secret'}}}, {'name': 'VLLM_EMBEDDING_MAX_TOKENS', 'value': '1024'}, {'name': 'VLLM_EMBEDDING_TLS_VERIFY', 'value': 'false'}, {'name': 'POSTGRES_HOST', 'value': 'vector-io-postgres-service'}, {'name': 'POSTGRES_PORT', 'value': '5432'}, {'name': 'POSTGRES_USER', 'valueFrom': {'secretKeyRef': {'key': 'postgres-user', 'name': 'ogx-distribution-secret'}}}, {'name': 'POSTGRES_PASSWORD', 'valueFrom': {'secretKeyRef': {'key': 'postgres-password', 'name': 'ogx-distribution-secret'}}}, {'name': 'POSTGRES_DB', 'value': 'ps_db'}, {'name': 'POSTGRES_TABLE_NAME', 'value': 'ogx_kvstore'}, {'name': 'MILVUS_ENDPOINT', 'value': 'http://vector-io-milvus-service:19530'}, {'name': 'MILVUS_TOKEN', 'valueFrom': {'secretKeyRef': {'key': 'milvus-token', 'name': 'vector-io-secret'}}}, {'name': 'MILVUS_CONSISTENCY_LEVEL', 'value': 'Bounded'}]}, 'replicas': 1, 'resources': {'limits': {'cpu': '2', 'memory': '2Gi'}, 'requests': {'cpu': '1', 'memory': '1Gi'}}, 'storage': {'mountPath': '/.ogx', 'size': '2Gi'}}}, 'status': {'availableReplicas': 1, 'conditions': [{'lastTransitionTime': '2026-07-05T07:28:44Z', 'message': 'Deployment is ready', 'reason': 'DeploymentReady', 'status': 'True', 'type': 'DeploymentReady'}, {'lastTransitionTime': '2026-07-05T07:28:44Z', 'message': 'Storage is ready', 'reason': 'StorageReady', 'status': 'True', 'type': 'StorageReady'}, {'lastTransitionTime': '2026-07-05T07:28:44Z', 'message': 'Service is ready', 'reason': 'ServiceReady', 'status': 'True', 'type': 'ServiceReady'}, {'lastTransitionTime': '2026-07-05T07:28:54Z', 'message': 'Health check passed', 'reason': 'HealthCheckPassed', 'status': 'True', 'type': 'HealthCheck'}], 'distributionConfig': {'activeDistribution': 'rh-dev', 'availableDistributions': {'rh-dev': 'registry.redhat.io/rhoai/odh-ogx-core-rhel9@sha256:3940a6ce94fe250d5d45634a73673bb5aa1a650e164cfecec1ad695fdd279242'}}, 'phase': 'Ready', 'serviceURL': 'http://ogx-server-500ac9b4-service.test-ogx-operator-09c9.svc.cluster.local:8321', 'version': {'lastUpdated': '2026-07-05T07:28:54Z', 'operatorVersion': '"0.10.0"'}}} [ocp_resources OGXServer] 2026-07-05T07:28:55.935469Z [info ] Wait until OGXServer ogx-server-500ac9b4 is deleted [ocp_resources OGXServer] 2026-07-05T07:28:55.935639Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:28:55.951352Z [info ] Elapsed time: 4.553794860839844e-05 [0:00:00.000046] [timeout_sampler] 2026-07-05T07:28:55.951570Z [info ] Delete Service vector-io-milvus-service [ocp_resources Service] 2026-07-05T07:28:55.978969Z [info ] Deleting {'kind': 'Service', 'apiVersion': 'v1', 'metadata': {'name': 'vector-io-milvus-service', 'namespace': 'test-ogx-operator-09c9', 'uid': '7ba72201-969e-43b5-902e-a5a9fa46209a', 'resourceVersion': '241447729', 'creationTimestamp': '2026-07-05T07:27:29Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:27:29Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:internalTrafficPolicy': {}, 'f:ports': {'.': {}, 'k:{"port":19530,"protocol":"TCP"}': {'.': {}, 'f:name': {}, 'f:port': {}, 'f:protocol': {}, 'f:targetPort': {}}}, 'f:selector': {}, 'f:sessionAffinity': {}, 'f:type': {}}}}]}, 'spec': {'ports': [{'name': 'grpc', 'protocol': 'TCP', 'port': 19530, 'targetPort': 19530}], 'selector': {'app': 'milvus-standalone'}, 'clusterIP': '172.30.22.109', 'clusterIPs': ['172.30.22.109'], 'type': 'ClusterIP', 'sessionAffinity': 'None', 'ipFamilies': ['IPv4'], 'ipFamilyPolicy': 'SingleStack', 'internalTrafficPolicy': 'Cluster'}, 'status': {'loadBalancer': {}}} [ocp_resources Service] 2026-07-05T07:28:56.011683Z [info ] Wait until Service vector-io-milvus-service is deleted [ocp_resources Service] 2026-07-05T07:28:56.011933Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:28:56.023502Z [info ] Elapsed time: 7.557868957519531e-05 [0:00:00.000076] [timeout_sampler] 2026-07-05T07:28:56.023733Z [info ] Delete Deployment vector-io-milvus-deployment [ocp_resources Deployment] 2026-07-05T07:28:56.050962Z [info ] Deleting {'kind': 'Deployment', 'apiVersion': 'apps/v1', 'metadata': {'name': 'vector-io-milvus-deployment', 'namespace': 'test-ogx-operator-09c9', 'uid': '8020b115-b38e-45f7-bb64-2ae27db83308', 'resourceVersion': '241447694', 'generation': 1, 'creationTimestamp': '2026-07-05T07:26:57Z', 'annotations': {'deployment.kubernetes.io/revision': '1'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'apps/v1', 'time': '2026-07-05T07:26:57Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:minReadySeconds': {}, 'f:progressDeadlineSeconds': {}, 'f:replicas': {}, 'f:revisionHistoryLimit': {}, 'f:selector': {}, 'f:strategy': {'f:type': {}}, 'f:template': {'f:metadata': {'f:labels': {'.': {}, 'f:app': {}}}, 'f:spec': {'f:containers': {'k:{"name":"milvus-standalone"}': {'.': {}, 'f:args': {}, 'f:env': {'.': {}, 'k:{"name":"COMMON_STORAGETYPE"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"DEPLOY_MODE"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"ETCD_ENDPOINTS"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"MINIO_ADDRESS"}': {'.': {}, 'f:name': {}}}, 'f:image': {}, 'f:imagePullPolicy': {}, 'f:name': {}, 'f:ports': {'.': {}, 'k:{"containerPort":19530,"protocol":"TCP"}': {'.': {}, 'f:containerPort': {}, 'f:protocol': {}}}, 'f:resources': {}, 'f:terminationMessagePath': {}, 'f:terminationMessagePolicy': {}, 'f:volumeMounts': {'.': {}, 'k:{"mountPath":"/var/lib/milvus"}': {'.': {}, 'f:mountPath': {}, 'f:name': {}}}}}, 'f:dnsPolicy': {}, 'f:restartPolicy': {}, 'f:schedulerName': {}, 'f:securityContext': {}, 'f:terminationGracePeriodSeconds': {}, 'f:volumes': {'.': {}, 'k:{"name":"milvus-data"}': {'.': {}, 'f:emptyDir': {}, 'f:name': {}}}}}}}}, {'manager': 'kube-controller-manager', 'operation': 'Update', 'apiVersion': 'apps/v1', 'time': '2026-07-05T07:27:28Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:deployment.kubernetes.io/revision': {}}}, 'f:status': {'f:availableReplicas': {}, 'f:conditions': {'.': {}, 'k:{"type":"Available"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:lastUpdateTime': {}, 'f:message': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"Progressing"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:lastUpdateTime': {}, 'f:message': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}}, 'f:observedGeneration': {}, 'f:readyReplicas': {}, 'f:replicas': {}, 'f:updatedReplicas': {}}}, 'subresource': 'status'}]}, 'spec': {'replicas': 1, 'selector': {'matchLabels': {'app': 'milvus-standalone'}}, 'template': {'metadata': {'creationTimestamp': None, 'labels': {'app': 'milvus-standalone'}}, 'spec': {'volumes': [{'name': 'milvus-data', 'emptyDir': {}}], 'containers': [{'name': 'milvus-standalone', 'image': 'quay.io/jgarciao/milvus@sha256:b7ae4fa7475f810837c91b61eb5aa39fd89f22348823fc76d9ce776a6ddc5844', 'args': ['milvus', 'run', 'standalone'], 'ports': [{'containerPort': 19530, 'protocol': 'TCP'}], 'env': [{'name': 'DEPLOY_MODE', 'value': 'standalone'}, {'name': 'ETCD_ENDPOINTS', 'value': 'vector-io-etcd-service:2379'}, {'name': 'MINIO_ADDRESS'}, {'name': 'COMMON_STORAGETYPE', 'value': 'local'}], 'resources': {}, 'volumeMounts': [{'name': 'milvus-data', 'mountPath': '/var/lib/milvus'}], 'terminationMessagePath': '/dev/termination-log', 'terminationMessagePolicy': 'File', 'imagePullPolicy': 'IfNotPresent'}], 'restartPolicy': 'Always', 'terminationGracePeriodSeconds': 30, 'dnsPolicy': 'ClusterFirst', 'securityContext': {}, 'schedulerName': 'default-scheduler'}}, 'strategy': {'type': 'Recreate'}, 'minReadySeconds': 5, 'revisionHistoryLimit': 10, 'progressDeadlineSeconds': 600}, 'status': {'observedGeneration': 1, 'replicas': 1, 'updatedReplicas': 1, 'readyReplicas': 1, 'availableReplicas': 1, 'conditions': [{'type': 'Available', 'status': 'True', 'lastUpdateTime': '2026-07-05T07:27:28Z', 'lastTransitionTime': '2026-07-05T07:27:28Z', 'reason': 'MinimumReplicasAvailable', 'message': 'Deployment has minimum availability.'}, {'type': 'Progressing', 'status': 'True', 'lastUpdateTime': '2026-07-05T07:27:28Z', 'lastTransitionTime': '2026-07-05T07:26:57Z', 'reason': 'NewReplicaSetAvailable', 'message': 'ReplicaSet "vector-io-milvus-deployment-6c56bf5769" has successfully progressed.'}]}} [ocp_resources Deployment] 2026-07-05T07:28:56.066455Z [info ] Wait until Deployment vector-io-milvus-deployment is deleted [ocp_resources Deployment] 2026-07-05T07:28:56.066585Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:28:56.077987Z [info ] Elapsed time: 7.677078247070312e-05 [0:00:00.000077] [timeout_sampler] 2026-07-05T07:28:56.078136Z [info ] Delete Secret vector-io-secret [ocp_resources Secret] 2026-07-05T07:28:56.104205Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'vector-io-secret', 'namespace': 'test-ogx-operator-09c9', 'uid': '11e4d329-b100-43f2-83b9-a1cce7181039', 'resourceVersion': '241446814', 'creationTimestamp': '2026-07-05T07:26:56Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:26:56Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:milvus-token': {}, 'f:pgvector-password': {}, 'f:pgvector-user': {}, 'f:qdrant-api-key': {}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-05T07:28:56.117659Z [info ] Wait until Secret vector-io-secret is deleted [ocp_resources Secret] 2026-07-05T07:28:56.117902Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:28:56.129678Z [info ] Elapsed time: 8.177757263183594e-05 [0:00:00.000082] [timeout_sampler] 2026-07-05T07:28:56.129916Z [info ] Delete Service vector-io-etcd-service [ocp_resources Service] 2026-07-05T07:28:56.153873Z [info ] Deleting {'kind': 'Service', 'apiVersion': 'v1', 'metadata': {'name': 'vector-io-etcd-service', 'namespace': 'test-ogx-operator-09c9', 'uid': 'd6308085-ffca-4834-ad4e-301f2fbab7b7', 'resourceVersion': '241446808', 'creationTimestamp': '2026-07-05T07:26:56Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:26:56Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:internalTrafficPolicy': {}, 'f:ports': {'.': {}, 'k:{"port":2379,"protocol":"TCP"}': {'.': {}, 'f:port': {}, 'f:protocol': {}, 'f:targetPort': {}}}, 'f:selector': {}, 'f:sessionAffinity': {}, 'f:type': {}}}}]}, 'spec': {'ports': [{'protocol': 'TCP', 'port': 2379, 'targetPort': 2379}], 'selector': {'app': 'etcd'}, 'clusterIP': '172.30.38.45', 'clusterIPs': ['172.30.38.45'], 'type': 'ClusterIP', 'sessionAffinity': 'None', 'ipFamilies': ['IPv4'], 'ipFamilyPolicy': 'SingleStack', 'internalTrafficPolicy': 'Cluster'}, 'status': {'loadBalancer': {}}} [ocp_resources Service] 2026-07-05T07:28:56.187124Z [info ] Wait until Service vector-io-etcd-service is deleted [ocp_resources Service] 2026-07-05T07:28:56.187257Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:28:56.198582Z [info ] Elapsed time: 5.364418029785156e-05 [0:00:00.000054] [timeout_sampler] 2026-07-05T07:28:56.198817Z [info ] Delete Deployment vector-io-etcd-deployment [ocp_resources Deployment] 2026-07-05T07:28:56.227044Z [info ] Deleting {'kind': 'Deployment', 'apiVersion': 'apps/v1', 'metadata': {'name': 'vector-io-etcd-deployment', 'namespace': 'test-ogx-operator-09c9', 'uid': '5606af54-6acf-493e-be22-46c7349ec930', 'resourceVersion': '241446791', 'generation': 1, 'creationTimestamp': '2026-07-05T07:26:52Z', 'annotations': {'deployment.kubernetes.io/revision': '1'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'apps/v1', 'time': '2026-07-05T07:26:52Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:progressDeadlineSeconds': {}, 'f:replicas': {}, 'f:revisionHistoryLimit': {}, 'f:selector': {}, 'f:strategy': {'f:type': {}}, 'f:template': {'f:metadata': {'f:labels': {'.': {}, 'f:app': {}}}, 'f:spec': {'f:containers': {'k:{"name":"etcd"}': {'.': {}, 'f:command': {}, 'f:env': {'.': {}, 'k:{"name":"ETCD_AUTO_COMPACTION_MODE"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"ETCD_AUTO_COMPACTION_RETENTION"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"ETCD_QUOTA_BACKEND_BYTES"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"ETCD_SNAPSHOT_COUNT"}': {'.': {}, 'f:name': {}, 'f:value': {}}}, 'f:image': {}, 'f:imagePullPolicy': {}, 'f:name': {}, 'f:ports': {'.': {}, 'k:{"containerPort":2379,"protocol":"TCP"}': {'.': {}, 'f:containerPort': {}, 'f:protocol': {}}}, 'f:resources': {}, 'f:terminationMessagePath': {}, 'f:terminationMessagePolicy': {}, 'f:volumeMounts': {'.': {}, 'k:{"mountPath":"/etcd"}': {'.': {}, 'f:mountPath': {}, 'f:name': {}}}}}, 'f:dnsPolicy': {}, 'f:restartPolicy': {}, 'f:schedulerName': {}, 'f:securityContext': {}, 'f:terminationGracePeriodSeconds': {}, 'f:volumes': {'.': {}, 'k:{"name":"etcd-data"}': {'.': {}, 'f:emptyDir': {}, 'f:name': {}}}}}}}}, {'manager': 'kube-controller-manager', 'operation': 'Update', 'apiVersion': 'apps/v1', 'time': '2026-07-05T07:26:56Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:deployment.kubernetes.io/revision': {}}}, 'f:status': {'f:availableReplicas': {}, 'f:conditions': {'.': {}, 'k:{"type":"Available"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:lastUpdateTime': {}, 'f:message': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"Progressing"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:lastUpdateTime': {}, 'f:message': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}}, 'f:observedGeneration': {}, 'f:readyReplicas': {}, 'f:replicas': {}, 'f:updatedReplicas': {}}}, 'subresource': 'status'}]}, 'spec': {'replicas': 1, 'selector': {'matchLabels': {'app': 'etcd'}}, 'template': {'metadata': {'creationTimestamp': None, 'labels': {'app': 'etcd'}}, 'spec': {'volumes': [{'name': 'etcd-data', 'emptyDir': {}}], 'containers': [{'name': 'etcd', 'image': 'quay.io/coreos/etcd@sha256:3397341272b9e0a6f44d7e3fc7c321c6efe6cbe82ce866b9b01d0c704bfc5bf3', 'command': ['etcd', '--advertise-client-urls=http://vector-io-etcd-service:2379', '--listen-client-urls=http://0.0.0.0:2379', '--data-dir=/etcd'], 'ports': [{'containerPort': 2379, 'protocol': 'TCP'}], 'env': [{'name': 'ETCD_AUTO_COMPACTION_MODE', 'value': 'revision'}, {'name': 'ETCD_AUTO_COMPACTION_RETENTION', 'value': '1000'}, {'name': 'ETCD_QUOTA_BACKEND_BYTES', 'value': '4294967296'}, {'name': 'ETCD_SNAPSHOT_COUNT', 'value': '50000'}], 'resources': {}, 'volumeMounts': [{'name': 'etcd-data', 'mountPath': '/etcd'}], 'terminationMessagePath': '/dev/termination-log', 'terminationMessagePolicy': 'File', 'imagePullPolicy': 'IfNotPresent'}], 'restartPolicy': 'Always', 'terminationGracePeriodSeconds': 30, 'dnsPolicy': 'ClusterFirst', 'securityContext': {}, 'schedulerName': 'default-scheduler'}}, 'strategy': {'type': 'Recreate'}, 'revisionHistoryLimit': 10, 'progressDeadlineSeconds': 600}, 'status': {'observedGeneration': 1, 'replicas': 1, 'updatedReplicas': 1, 'readyReplicas': 1, 'availableReplicas': 1, 'conditions': [{'type': 'Available', 'status': 'True', 'lastUpdateTime': '2026-07-05T07:26:56Z', 'lastTransitionTime': '2026-07-05T07:26:56Z', 'reason': 'MinimumReplicasAvailable', 'message': 'Deployment has minimum availability.'}, {'type': 'Progressing', 'status': 'True', 'lastUpdateTime': '2026-07-05T07:26:56Z', 'lastTransitionTime': '2026-07-05T07:26:52Z', 'reason': 'NewReplicaSetAvailable', 'message': 'ReplicaSet "vector-io-etcd-deployment-5784dbfbb6" has successfully progressed.'}]}} [ocp_resources Deployment] 2026-07-05T07:28:56.242485Z [info ] Wait until Deployment vector-io-etcd-deployment is deleted [ocp_resources Deployment] 2026-07-05T07:28:56.242670Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:28:56.265868Z [info ] Elapsed time: 8.130073547363281e-05 [0:00:00.000081] [timeout_sampler] 2026-07-05T07:28:56.266014Z [info ] Delete Service vector-io-postgres-service [ocp_resources Service] 2026-07-05T07:28:56.290935Z [info ] Deleting {'kind': 'Service', 'apiVersion': 'v1', 'metadata': {'name': 'vector-io-postgres-service', 'namespace': 'test-ogx-operator-09c9', 'uid': 'a407b9b8-0f18-4fb7-96ad-3119a6ca3184', 'resourceVersion': '241446663', 'creationTimestamp': '2026-07-05T07:26:52Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:26:52Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:internalTrafficPolicy': {}, 'f:ports': {'.': {}, 'k:{"port":5432,"protocol":"TCP"}': {'.': {}, 'f:port': {}, 'f:protocol': {}, 'f:targetPort': {}}}, 'f:selector': {}, 'f:sessionAffinity': {}, 'f:type': {}}}}]}, 'spec': {'ports': [{'protocol': 'TCP', 'port': 5432, 'targetPort': 5432}], 'selector': {'app': 'postgres'}, 'clusterIP': '172.30.252.39', 'clusterIPs': ['172.30.252.39'], 'type': 'ClusterIP', 'sessionAffinity': 'None', 'ipFamilies': ['IPv4'], 'ipFamilyPolicy': 'SingleStack', 'internalTrafficPolicy': 'Cluster'}, 'status': {'loadBalancer': {}}} [ocp_resources Service] 2026-07-05T07:28:56.323515Z [info ] Wait until Service vector-io-postgres-service is deleted [ocp_resources Service] 2026-07-05T07:28:56.323668Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:28:56.338585Z [info ] Elapsed time: 8.106231689453125e-05 [0:00:00.000081] [timeout_sampler] 2026-07-05T07:28:56.338819Z [info ] Delete Deployment vector-io-postgres-deployment [ocp_resources Deployment] 2026-07-05T07:28:56.368063Z [info ] Deleting {'kind': 'Deployment', 'apiVersion': 'apps/v1', 'metadata': {'name': 'vector-io-postgres-deployment', 'namespace': 'test-ogx-operator-09c9', 'uid': 'fb0eb647-f647-471a-8a9e-679e02389dad', 'resourceVersion': '241446659', 'generation': 1, 'creationTimestamp': '2026-07-05T07:26:46Z', 'annotations': {'deployment.kubernetes.io/revision': '1'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'apps/v1', 'time': '2026-07-05T07:26:46Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:minReadySeconds': {}, 'f:progressDeadlineSeconds': {}, 'f:replicas': {}, 'f:revisionHistoryLimit': {}, 'f:selector': {}, 'f:strategy': {'f:type': {}}, 'f:template': {'f:metadata': {'f:labels': {'.': {}, 'f:app': {}}}, 'f:spec': {'f:containers': {'k:{"name":"postgres"}': {'.': {}, 'f:env': {'.': {}, 'k:{"name":"POSTGRESQL_DATABASE"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"POSTGRESQL_PASSWORD"}': {'.': {}, 'f:name': {}, 'f:valueFrom': {'.': {}, 'f:secretKeyRef': {}}}, 'k:{"name":"POSTGRESQL_USER"}': {'.': {}, 'f:name': {}, 'f:valueFrom': {'.': {}, 'f:secretKeyRef': {}}}}, 'f:image': {}, 'f:imagePullPolicy': {}, 'f:name': {}, 'f:ports': {'.': {}, 'k:{"containerPort":5432,"protocol":"TCP"}': {'.': {}, 'f:containerPort': {}, 'f:protocol': {}}}, 'f:resources': {}, 'f:terminationMessagePath': {}, 'f:terminationMessagePolicy': {}, 'f:volumeMounts': {'.': {}, 'k:{"mountPath":"/var/lib/pgsql/data"}': {'.': {}, 'f:mountPath': {}, 'f:name': {}}}}}, 'f:dnsPolicy': {}, 'f:restartPolicy': {}, 'f:schedulerName': {}, 'f:securityContext': {}, 'f:terminationGracePeriodSeconds': {}, 'f:volumes': {'.': {}, 'k:{"name":"postgresdata"}': {'.': {}, 'f:emptyDir': {}, 'f:name': {}}}}}}}}, {'manager': 'kube-controller-manager', 'operation': 'Update', 'apiVersion': 'apps/v1', 'time': '2026-07-05T07:26:52Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:deployment.kubernetes.io/revision': {}}}, 'f:status': {'f:availableReplicas': {}, 'f:conditions': {'.': {}, 'k:{"type":"Available"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:lastUpdateTime': {}, 'f:message': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"Progressing"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:lastUpdateTime': {}, 'f:message': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}}, 'f:observedGeneration': {}, 'f:readyReplicas': {}, 'f:replicas': {}, 'f:updatedReplicas': {}}}, 'subresource': 'status'}]}, 'spec': {'replicas': 1, 'selector': {'matchLabels': {'app': 'postgres'}}, 'template': {'metadata': {'creationTimestamp': None, 'labels': {'app': 'postgres'}}, 'spec': {'volumes': [{'name': 'postgresdata', 'emptyDir': {}}], 'containers': [{'name': 'postgres', 'image': 'registry.redhat.io/rhel9/postgresql-15@sha256:90ec347a35ab8a5d530c8d09f5347b13cc71df04f3b994bfa8b1a409b1171d59', 'ports': [{'containerPort': 5432, 'protocol': 'TCP'}], 'env': [{'name': 'POSTGRESQL_DATABASE', 'value': 'ps_db'}, {'name': 'POSTGRESQL_USER', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'postgres-user'}}}, {'name': 'POSTGRESQL_PASSWORD', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'postgres-password'}}}], 'resources': {}, 'volumeMounts': [{'name': 'postgresdata', 'mountPath': '/var/lib/pgsql/data'}], 'terminationMessagePath': '/dev/termination-log', 'terminationMessagePolicy': 'File', 'imagePullPolicy': 'IfNotPresent'}], 'restartPolicy': 'Always', 'terminationGracePeriodSeconds': 30, 'dnsPolicy': 'ClusterFirst', 'securityContext': {}, 'schedulerName': 'default-scheduler'}}, 'strategy': {'type': 'Recreate'}, 'minReadySeconds': 5, 'revisionHistoryLimit': 10, 'progressDeadlineSeconds': 600}, 'status': {'observedGeneration': 1, 'replicas': 1, 'updatedReplicas': 1, 'readyReplicas': 1, 'availableReplicas': 1, 'conditions': [{'type': 'Available', 'status': 'True', 'lastUpdateTime': '2026-07-05T07:26:52Z', 'lastTransitionTime': '2026-07-05T07:26:52Z', 'reason': 'MinimumReplicasAvailable', 'message': 'Deployment has minimum availability.'}, {'type': 'Progressing', 'status': 'True', 'lastUpdateTime': '2026-07-05T07:26:52Z', 'lastTransitionTime': '2026-07-05T07:26:46Z', 'reason': 'NewReplicaSetAvailable', 'message': 'ReplicaSet "vector-io-postgres-deployment-5db8dfbb79" has successfully progressed.'}]}} [ocp_resources Deployment] 2026-07-05T07:28:56.382614Z [info ] Wait until Deployment vector-io-postgres-deployment is deleted [ocp_resources Deployment] 2026-07-05T07:28:56.383025Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:28:56.395138Z [info ] Elapsed time: 9.560585021972656e-05 [0:00:00.000096] [timeout_sampler] 2026-07-05T07:28:56.395296Z [info ] Delete Secret ogx-distribution-secret [ocp_resources Secret] 2026-07-05T07:28:56.421424Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'ogx-distribution-secret', 'namespace': 'test-ogx-operator-09c9', 'uid': '93fc9a33-a9ae-43b9-8d0d-e0a216dec5c5', 'resourceVersion': '241446463', 'creationTimestamp': '2026-07-05T07:26:46Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:26:46Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:aws-access-key-id': {}, 'f:aws-secret-access-key': {}, 'f:postgres-password': {}, 'f:postgres-user': {}, 'f:vllm-api-token': {}, 'f:vllm-embedding-api-token': {}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-05T07:28:56.439550Z [info ] Wait until Secret ogx-distribution-secret is deleted [ocp_resources Secret] 2026-07-05T07:28:56.439734Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:28:56.452452Z [info ] Elapsed time: 5.435943603515625e-05 [0:00:00.000054] [timeout_sampler] 2026-07-05T07:28:56.575974Z [info ] Delete Project test-ogx-operator-09c9 [ocp_resources Project] 2026-07-05T07:28:56.611034Z [info ] Deleting {'kind': 'Project', 'apiVersion': 'project.openshift.io/v1', 'metadata': {'name': 'test-ogx-operator-09c9', 'uid': '18cf75a2-397d-4a7c-86a0-dbe7f94735ce', 'resourceVersion': '241450116', 'creationTimestamp': '2026-07-05T07:26:46Z', 'labels': {'kubernetes.io/metadata.name': 'test-ogx-operator-09c9', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/description': '', 'openshift.io/display-name': '', 'openshift.io/requester': 'system:serviceaccount:kube-system:olminstall-cluster-admin', 'openshift.io/sa.scc.mcs': 's0:c31,c25', 'openshift.io/sa.scc.supplemental-groups': '1000980000/10000', 'openshift.io/sa.scc.uid-range': '1000980000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-07-05T07:28:55Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:26:46Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}, {'manager': 'openshift-apiserver', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:26:46Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/description': {}, 'f:openshift.io/display-name': {}, 'f:openshift.io/requester': {}}, 'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'openshift-controller-manager', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:26:46Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:finalizers': {}}}, 'subresource': 'finalize'}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Project] 2026-07-05T07:28:56.629384Z [info ] Wait until Project test-ogx-operator-09c9 is deleted [ocp_resources Project] 2026-07-05T07:28:56.629548Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] _ 1 of 4 completed, 1 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/ogx/vector_io/test_vector_stores.py::TestOgxVectorStores::test_vector_stores_file_upload[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] 2026-07-05T07:29:49.461800Z [info ] Elapsed time: 52.81177377700806 [0:00:52.811774] [timeout_sampler]  test_vector_stores_file_upload[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS]  -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-07-05T07:29:49.462359Z [info ] Executing class fixture: unprivileged_model_namespace [conftest] 2026-07-05T07:29:49.462479Z [info ] kind: ProjectRequest api version: project.openshift.io/v1 [ocp_resources.resource] 2026-07-05T07:29:49.462593Z [info ] Create ProjectRequest test-ogx-vector-stores-f99f [ocp_resources ProjectRequest] 2026-07-05T07:29:49.462727Z [info ] Posting {'apiVersion': 'project.openshift.io/v1', 'kind': 'ProjectRequest', 'metadata': {'name': 'test-ogx-vector-stores-f99f'}} [ocp_resources ProjectRequest] 2026-07-05T07:29:49.547661Z [info ] kind: Project api version: project.openshift.io/v1 [ocp_resources.resource] 2026-07-05T07:29:49.547847Z [info ] Wait for Project test-ogx-vector-stores-f99f status to be Active [ocp_resources Project] 2026-07-05T07:29:49.547992Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-07-05T07:29:49.563931Z [info ] Status of Project test-ogx-vector-stores-f99f is Active [ocp_resources Project] 2026-07-05T07:29:49.564085Z [info ] Elapsed time: 3.218650817871094e-05 [0:00:00.000032] [timeout_sampler] 2026-07-05T07:29:49.564206Z [info ] Executing class fixture: distribution_name [conftest] 2026-07-05T07:29:49.564331Z [info ] Executing class fixture: enabled_ogx_operator [conftest] 2026-07-05T07:29:49.591717Z [warning ] Component ogx was already set to managementState Managed [utilities.data_science_cluster_utils] name=utilities.data_science_cluster_utils 2026-07-05T07:29:49.594021Z [info ] Executing class fixture: vector_io_provider_deployment_config_factory [conftest] 2026-07-05T07:29:49.594155Z [info ] Executing class fixture: files_provider_config_factory [conftest] 2026-07-05T07:29:49.594276Z [info ] Executing class fixture: is_disconnected_cluster [conftest] 2026-07-05T07:29:49.594395Z [info ] Executing class fixture: ogx_server_secret [conftest] 2026-07-05T07:29:49.594516Z [info ] Create Secret ogx-distribution-secret [ocp_resources Secret] 2026-07-05T07:29:49.594636Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'ogx-distribution-secret', 'namespace': 'test-ogx-vector-stores-f99f'}, 'stringData': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-05T07:29:49.619306Z [info ] Executing class fixture: postgres_deployment [conftest] 2026-07-05T07:29:49.619464Z [info ] kind: Deployment api version: apps/v1 [ocp_resources.resource] 2026-07-05T07:29:49.619610Z [info ] Create Deployment vector-io-postgres-deployment [ocp_resources Deployment] 2026-07-05T07:29:49.619823Z [info ] Posting {'apiVersion': 'apps/v1', 'kind': 'Deployment', 'metadata': {'name': 'vector-io-postgres-deployment', 'namespace': 'test-ogx-vector-stores-f99f'}, 'spec': {'selector': {'matchLabels': {'app': 'postgres'}}, 'template': {'metadata': {'labels': {'app': 'postgres'}}, 'spec': {'containers': [{'name': 'postgres', 'image': 'registry.redhat.io/rhel9/postgresql-15@sha256:90ec347a35ab8a5d530c8d09f5347b13cc71df04f3b994bfa8b1a409b1171d59', 'ports': [{'containerPort': 5432}], 'env': [{'name': 'POSTGRESQL_DATABASE', 'value': 'ps_db'}, {'name': 'POSTGRESQL_USER', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'postgres-user'}}}, {'name': 'POSTGRESQL_PASSWORD', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'postgres-password'}}}], 'volumeMounts': [{'name': 'postgresdata', 'mountPath': '/var/lib/pgsql/data'}]}], 'volumes': [{'name': 'postgresdata', 'emptyDir': {}}]}}, 'minReadySeconds': 5, 'replicas': 1, 'strategy': {'type': 'Recreate'}}} [ocp_resources Deployment] 2026-07-05T07:29:49.666414Z [info ] Wait for Deployment vector-io-postgres-deployment to be deployed: True [ocp_resources Deployment] 2026-07-05T07:29:49.666568Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-07-05T07:29:49.677502Z [info ] Elapsed time: 8.273124694824219e-05 [0:00:00.000083] [timeout_sampler] 2026-07-05T07:29:49.677665Z [info ] Waiting for 239.98875069618225 seconds [0:03:59.988751], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-07-05T07:29:55.764540Z [info ] Elapsed time: 6.073714733123779 [0:00:06.073715] [timeout_sampler] 2026-07-05T07:29:55.764716Z [info ] Executing class fixture: postgres_service [conftest] 2026-07-05T07:29:55.764848Z [info ] Create Service vector-io-postgres-service [ocp_resources Service] 2026-07-05T07:29:55.764971Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'vector-io-postgres-service', 'namespace': 'test-ogx-vector-stores-f99f'}, 'spec': {'ports': [{'port': 5432, 'targetPort': 5432}], 'selector': {'app': 'postgres'}}} [ocp_resources Service] 2026-07-05T07:29:55.801004Z [info ] Wait until Service vector-io-postgres-service is created [ocp_resources Service] 2026-07-05T07:29:55.801161Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-07-05T07:29:55.814322Z [info ] Elapsed time: 8.440017700195312e-05 [0:00:00.000084] [timeout_sampler] 2026-07-05T07:29:55.814487Z [info ] Executing class fixture: ogx_server [conftest] 2026-07-05T07:29:55.814623Z [info ] Executing session fixture: ci_s3_bucket_name [conftest] 2026-07-05T07:29:55.814787Z [info ] Executing session fixture: ci_s3_bucket_region [conftest] 2026-07-05T07:29:55.814912Z [info ] Executing session fixture: ci_s3_bucket_endpoint [conftest] 2026-07-05T07:29:55.815036Z [info ] Executing class fixture: etcd_deployment [conftest] 2026-07-05T07:29:55.815153Z [info ] Create Deployment vector-io-etcd-deployment [ocp_resources Deployment] 2026-07-05T07:29:55.815302Z [info ] Posting {'apiVersion': 'apps/v1', 'kind': 'Deployment', 'metadata': {'name': 'vector-io-etcd-deployment', 'namespace': 'test-ogx-vector-stores-f99f'}, 'spec': {'selector': {'matchLabels': {'app': 'etcd'}}, 'template': {'metadata': {'labels': {'app': 'etcd'}}, 'spec': {'containers': [{'name': 'etcd', 'image': 'quay.io/coreos/etcd@sha256:3397341272b9e0a6f44d7e3fc7c321c6efe6cbe82ce866b9b01d0c704bfc5bf3', 'command': ['etcd', '--advertise-client-urls=http://vector-io-etcd-service:2379', '--listen-client-urls=http://0.0.0.0:2379', '--data-dir=/etcd'], 'ports': [{'containerPort': 2379}], 'volumeMounts': [{'name': 'etcd-data', 'mountPath': '/etcd'}], 'env': [{'name': 'ETCD_AUTO_COMPACTION_MODE', 'value': 'revision'}, {'name': 'ETCD_AUTO_COMPACTION_RETENTION', 'value': '1000'}, {'name': 'ETCD_QUOTA_BACKEND_BYTES', 'value': '4294967296'}, {'name': 'ETCD_SNAPSHOT_COUNT', 'value': '50000'}]}], 'volumes': [{'name': 'etcd-data', 'emptyDir': {}}]}}, 'replicas': 1, 'strategy': {'type': 'Recreate'}}} [ocp_resources Deployment] 2026-07-05T07:29:55.854876Z [info ] Wait for Deployment vector-io-etcd-deployment to be deployed: True [ocp_resources Deployment] 2026-07-05T07:29:55.855008Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-07-05T07:29:55.869905Z [info ] Elapsed time: 6.842613220214844e-05 [0:00:00.000068] [timeout_sampler] 2026-07-05T07:29:55.870059Z [info ] Waiting for 119.98495650291443 seconds [0:01:59.984957], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-07-05T07:29:57.918493Z [info ] Elapsed time: 2.028374671936035 [0:00:02.028375] [timeout_sampler] 2026-07-05T07:29:57.918669Z [info ] Executing class fixture: etcd_service [conftest] 2026-07-05T07:29:57.918825Z [info ] Create Service vector-io-etcd-service [ocp_resources Service] 2026-07-05T07:29:57.918944Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'vector-io-etcd-service', 'namespace': 'test-ogx-vector-stores-f99f'}, 'spec': {'ports': [{'port': 2379, 'targetPort': 2379}], 'selector': {'app': 'etcd'}}} [ocp_resources Service] 2026-07-05T07:29:57.988888Z [info ] Wait until Service vector-io-etcd-service is created [ocp_resources Service] 2026-07-05T07:29:57.989052Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-07-05T07:29:58.004095Z [info ] Elapsed time: 7.081031799316406e-05 [0:00:00.000071] [timeout_sampler] 2026-07-05T07:29:58.004246Z [info ] Executing class fixture: vector_io_secret [conftest] 2026-07-05T07:29:58.004396Z [info ] Create Secret vector-io-secret [ocp_resources Secret] 2026-07-05T07:29:58.004550Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'vector-io-secret', 'namespace': 'test-ogx-vector-stores-f99f'}, 'stringData': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-05T07:29:58.038678Z [info ] Executing class fixture: remote_milvus_deployment [conftest] 2026-07-05T07:29:58.038897Z [info ] Create Deployment vector-io-milvus-deployment [ocp_resources Deployment] 2026-07-05T07:29:58.039073Z [info ] Posting {'apiVersion': 'apps/v1', 'kind': 'Deployment', 'metadata': {'name': 'vector-io-milvus-deployment', 'namespace': 'test-ogx-vector-stores-f99f'}, 'spec': {'selector': {'matchLabels': {'app': 'milvus-standalone'}}, 'template': {'metadata': {'labels': {'app': 'milvus-standalone'}}, 'spec': {'containers': [{'name': 'milvus-standalone', 'image': 'quay.io/jgarciao/milvus@sha256:b7ae4fa7475f810837c91b61eb5aa39fd89f22348823fc76d9ce776a6ddc5844', 'args': ['milvus', 'run', 'standalone'], 'ports': [{'containerPort': 19530, 'protocol': 'TCP'}], 'volumeMounts': [{'name': 'milvus-data', 'mountPath': '/var/lib/milvus'}], 'env': [{'name': 'DEPLOY_MODE', 'value': 'standalone'}, {'name': 'ETCD_ENDPOINTS', 'value': 'vector-io-etcd-service:2379'}, {'name': 'MINIO_ADDRESS', 'value': ''}, {'name': 'COMMON_STORAGETYPE', 'value': 'local'}]}], 'volumes': [{'name': 'milvus-data', 'emptyDir': {}}]}}, 'minReadySeconds': 5, 'replicas': 1, 'strategy': {'type': 'Recreate'}}} [ocp_resources Deployment] 2026-07-05T07:29:58.075090Z [info ] Wait for Deployment vector-io-milvus-deployment to be deployed: True [ocp_resources Deployment] 2026-07-05T07:29:58.075244Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-07-05T07:29:58.090208Z [info ] Elapsed time: 7.581710815429688e-05 [0:00:00.000076] [timeout_sampler] 2026-07-05T07:29:58.090358Z [info ] Waiting for 239.98469853401184 seconds [0:03:59.984699], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-07-05T07:30:05.196281Z [info ] Elapsed time: 7.0936760902404785 [0:00:07.093676] [timeout_sampler] 2026-07-05T07:30:05.196539Z [info ] Executing class fixture: milvus_service [conftest] 2026-07-05T07:30:05.196680Z [info ] Create Service vector-io-milvus-service [ocp_resources Service] 2026-07-05T07:30:05.196870Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'vector-io-milvus-service', 'namespace': 'test-ogx-vector-stores-f99f'}, 'spec': {'ports': [{'name': 'grpc', 'port': 19530, 'targetPort': 19530}], 'selector': {'app': 'milvus-standalone'}}} [ocp_resources Service] 2026-07-05T07:30:05.228028Z [info ] Wait until Service vector-io-milvus-service is created [ocp_resources Service] 2026-07-05T07:30:05.228207Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-07-05T07:30:05.242506Z [info ] Elapsed time: 4.1961669921875e-05 [0:00:00.000042] [timeout_sampler] 2026-07-05T07:30:05.242753Z [info ] kind: OGXServer api version: ogx.io/v1beta1 --- [DuplicateFilter: Last log `kind: Deployment api version: apps/v1` repeated 2 times] [ocp_resources.resource] 2026-07-05T07:30:05.242911Z [info ] Create OGXServer ogx-server-7f26f05f [ocp_resources OGXServer] 2026-07-05T07:30:05.243076Z [info ] Posting {'apiVersion': 'ogx.io/v1beta1', 'kind': 'OGXServer', 'metadata': {'name': 'ogx-server-7f26f05f', 'namespace': 'test-ogx-vector-stores-f99f'}, 'spec': {'distribution': {'name': 'rh-dev'}, 'workload': {'resources': {'requests': {'cpu': '1', 'memory': '1Gi'}, 'limits': {'cpu': '2', 'memory': '2Gi'}}, 'overrides': {'env': [{'name': 'INFERENCE_MODEL', 'value': 'Qwen3.6-35B-A3B'}, {'name': 'VLLM_API_TOKEN', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'vllm-api-token'}}}, {'name': 'VLLM_URL', 'value': 'https://litellm-litemaas.apps.prod.rhoai.rh-aiservices-bu.com/v1'}, {'name': 'VLLM_TLS_VERIFY', 'value': 'false'}, {'name': 'VLLM_MAX_TOKENS', 'value': '80000'}, {'name': 'EMBEDDING_MODEL', 'value': 'Nomic-embed-text-v2-moe'}, {'name': 'EMBEDDING_PROVIDER_MODEL_ID', 'value': 'Nomic-embed-text-v2-moe'}, {'name': 'VLLM_EMBEDDING_URL', 'value': 'https://litellm-litemaas.apps.prod.rhoai.rh-aiservices-bu.com/v1'}, {'name': 'VLLM_EMBEDDING_API_TOKEN', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'vllm-embedding-api-token'}}}, {'name': 'VLLM_EMBEDDING_MAX_TOKENS', 'value': '1024'}, {'name': 'VLLM_EMBEDDING_TLS_VERIFY', 'value': 'false'}, {'name': 'POSTGRES_HOST', 'value': 'vector-io-postgres-service'}, {'name': 'POSTGRES_PORT', 'value': '5432'}, {'name': 'POSTGRES_USER', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'postgres-user'}}}, {'name': 'POSTGRES_PASSWORD', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'postgres-password'}}}, {'name': 'POSTGRES_DB', 'value': 'ps_db'}, {'name': 'POSTGRES_TABLE_NAME', 'value': 'ogx_kvstore'}, {'name': 'ENABLE_S3', 'value': 's3'}, {'name': 'S3_BUCKET_NAME', 'value': 'opendatahub-tests'}, {'name': 'AWS_DEFAULT_REGION', 'value': 'us-east-1'}, {'name': 'S3_ENDPOINT_URL', 'value': 'https://s3.us-east-1.amazonaws.com/'}, {'name': 'AWS_ACCESS_KEY_ID', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'aws-access-key-id'}}}, {'name': 'AWS_SECRET_ACCESS_KEY', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'aws-secret-access-key'}}}, {'name': 'S3_AUTO_CREATE_BUCKET', 'value': 'true'}, {'name': 'MILVUS_ENDPOINT', 'value': 'http://vector-io-milvus-service:19530'}, {'name': 'MILVUS_TOKEN', 'valueFrom': {'secretKeyRef': {'name': 'vector-io-secret', 'key': 'milvus-token'}}}, {'name': 'MILVUS_CONSISTENCY_LEVEL', 'value': 'Bounded'}]}}, 'network': {'policy': {'ingress': [{'from': [{'namespaceSelector': {'matchLabels': {'kubernetes.io/metadata.name': 'openshift-ingress'}}}], 'ports': [{'protocol': 'TCP', 'port': 8321}]}]}}}} [ocp_resources OGXServer] 2026-07-05T07:30:05.277687Z [info ] Wait until OGXServer ogx-server-7f26f05f is created [ocp_resources OGXServer] 2026-07-05T07:30:05.277892Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-07-05T07:30:05.289649Z [info ] Elapsed time: 6.985664367675781e-05 [0:00:00.000070] [timeout_sampler] 2026-07-05T07:30:05.289884Z [info ] Wait for OGXServer ogx-server-7f26f05f status to be Ready [ocp_resources OGXServer] 2026-07-05T07:30:05.290027Z [info ] Waiting for 600 seconds [0:10:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-07-05T07:30:06.314426Z [info ] Status of OGXServer ogx-server-7f26f05f is Initializing [ocp_resources OGXServer] 2026-07-05T07:30:36.703739Z [info ] Status of OGXServer ogx-server-7f26f05f is Ready [ocp_resources OGXServer] 2026-07-05T07:30:36.704021Z [info ] Elapsed time: 31.400065660476685 [0:00:31.400066] [timeout_sampler] 2026-07-05T07:30:36.704178Z [info ] Executing class fixture: ogx_server_deployment [conftest] 2026-07-05T07:30:36.704332Z [info ] kind: Deployment api version: apps/v1 [ocp_resources.resource] 2026-07-05T07:30:36.704476Z [info ] Wait until Deployment ogx-server-7f26f05f is created [ocp_resources Deployment] 2026-07-05T07:30:36.704603Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-07-05T07:30:36.725948Z [info ] Elapsed time: 5.2928924560546875e-05 [0:00:00.000053] [timeout_sampler] 2026-07-05T07:30:36.726108Z [info ] Wait for Deployment ogx-server-7f26f05f to be deployed: True [ocp_resources Deployment] 2026-07-05T07:30:36.726241Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-07-05T07:30:36.737640Z [info ] Elapsed time: 2.6464462280273438e-05 [0:00:00.000026] [timeout_sampler] 2026-07-05T07:30:36.737893Z [info ] Waiting for 239.9881784915924 seconds [0:03:59.988178], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-07-05T07:30:36.749937Z [info ] Elapsed time: 2.9802322387695312e-05 [0:00:00.000030] [timeout_sampler] 2026-07-05T07:30:36.750098Z [info ] Waiting for 240 seconds [0:04:00], retry every 5 seconds. (Function: tests.ogx.utils.wait_for_unique_ogx_pod Kwargs: {'client': , 'namespace': 'test-ogx-vector-stores-f99f'}) [timeout_sampler] 2026-07-05T07:30:36.764456Z [info ] Elapsed time: 3.409385681152344e-05 [0:00:00.000034] [timeout_sampler] 2026-07-05T07:30:36.764599Z [info ] Executing class fixture: ogx_test_route [conftest] 2026-07-05T07:30:36.764752Z [info ] kind: Route api version: route.openshift.io/v1 [ocp_resources.resource] 2026-07-05T07:30:36.764880Z [info ] Create Route ogx-7168f58760ec  [ocp_resources Route] 2026-07-05T07:30:36.765005Z [info ] Posting {'apiVersion': 'route.openshift.io/v1', 'kind': 'Route', 'metadata': {'name': 'ogx-7168f58760ec', 'namespace': 'test-ogx-vector-stores-f99f'}, 'spec': {'to': {'kind': 'Service', 'name': 'ogx-server-7f26f05f-service'}}} [ocp_resources Route] 2026-07-05T07:30:36.796663Z [info ] Wait until Route ogx-7168f58760ec is created [ocp_resources Route] 2026-07-05T07:30:36.796909Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-07-05T07:30:36.812503Z [info ] Elapsed time: 7.033348083496094e-05 [0:00:00.000070] [timeout_sampler] 2026-07-05T07:30:36.812663Z [info ] ResourceEdits: Updating data for resource Route ogx-7168f58760ec [ocp_resources.resource] 2026-07-05T07:30:36.812885Z [info ] Update Route ogx-7168f58760ec: {'spec': {'tls': {'termination': 'edge', 'insecureEdgeTerminationPolicy': 'Redirect'}}, 'metadata': {'annotations': {'haproxy.router.openshift.io/timeout': '10m'}, 'name': 'ogx-7168f58760ec'}} [ocp_resources Route] 2026-07-05T07:30:36.832054Z [info ] Wait until Route ogx-7168f58760ec is created [ocp_resources Route] 2026-07-05T07:30:36.832209Z [info ] Waiting for 60 seconds [0:01:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-07-05T07:30:36.844435Z [info ] Elapsed time: 3.337860107421875e-05 [0:00:00.000033] [timeout_sampler] 2026-07-05T07:30:36.844573Z [info ] Executing class fixture: ogx_client [conftest] 2026-07-05T07:30:36.861095Z [info ] Waiting for 90 seconds [0:01:30], retry every 5 seconds. (Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': }) [timeout_sampler] 2026-07-05T07:30:37.003028Z [info ] HTTP Request: GET https://ogx-7168f58760ec-test-ogx-vector-stores-f99f.apps.ods-qe-psi-23.osp.rh-ods.com/v1/health "HTTP/1.0 503 Service Unavailable" [httpx] 2026-07-05T07:30:37.003393Z [info ] Retrying request to /v1/health in 0.497202 seconds [ogx_client._base_client] 2026-07-05T07:30:37.546792Z [info ] HTTP Request: GET https://ogx-7168f58760ec-test-ogx-vector-stores-f99f.apps.ods-qe-psi-23.osp.rh-ods.com/v1/health "HTTP/1.0 503 Service Unavailable" [httpx] 2026-07-05T07:30:37.547098Z [info ] Retrying request to /v1/health in 0.994704 seconds [ogx_client._base_client] 2026-07-05T07:30:38.584013Z [info ] HTTP Request: GET https://ogx-7168f58760ec-test-ogx-vector-stores-f99f.apps.ods-qe-psi-23.osp.rh-ods.com/v1/health "HTTP/1.0 503 Service Unavailable" [httpx] 2026-07-05T07:30:38.584367Z [info ] Retrying request to /v1/health in 1.936081 seconds [ogx_client._base_client] 2026-07-05T07:30:40.564349Z [info ] HTTP Request: GET https://ogx-7168f58760ec-test-ogx-vector-stores-f99f.apps.ods-qe-psi-23.osp.rh-ods.com/v1/health "HTTP/1.0 503 Service Unavailable" [httpx] 2026-07-05T07:31:05.608834Z [info ] HTTP Request: GET https://ogx-7168f58760ec-test-ogx-vector-stores-f99f.apps.ods-qe-psi-23.osp.rh-ods.com/v1/health "HTTP/1.0 503 Service Unavailable" [httpx] 2026-07-05T07:31:05.609188Z [info ] Retrying request to /v1/health in 0.424544 seconds [ogx_client._base_client] 2026-07-05T07:31:26.080511Z [info ] HTTP Request: GET https://ogx-7168f58760ec-test-ogx-vector-stores-f99f.apps.ods-qe-psi-23.osp.rh-ods.com/v1/health "HTTP/1.0 503 Service Unavailable" [httpx] 2026-07-05T07:31:26.080921Z [info ] Retrying request to /v1/health in 0.797204 seconds [ogx_client._base_client] 2026-07-05T07:31:46.931797Z [info ] HTTP Request: GET https://ogx-7168f58760ec-test-ogx-vector-stores-f99f.apps.ods-qe-psi-23.osp.rh-ods.com/v1/health "HTTP/1.0 503 Service Unavailable" [httpx] 2026-07-05T07:31:46.932137Z [info ] Retrying request to /v1/health in 1.532057 seconds [ogx_client._base_client] 2026-07-05T07:32:08.513008Z [info ] HTTP Request: GET https://ogx-7168f58760ec-test-ogx-vector-stores-f99f.apps.ods-qe-psi-23.osp.rh-ods.com/v1/health "HTTP/1.0 503 Service Unavailable" [httpx] ERROR _ ERROR at setup of TestOgxVectorStores.test_vector_stores_file_upload[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1110, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 728, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 908, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/ogx/conftest.py", line 292, in ogx_client wait_for_ogx_client_ready(client=client) ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 280, in wrapper for sample in TimeoutSampler( ~~~~~~~~~~~~~~^ func=func, ^^^^^^^^^^ ...<7 lines>... **kwargs, ^^^^^^^^^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 90 Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': } Last exception: N/A. _ 2 of 4 completed, 1 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 1 Error, 0 ReRun _ tests/ogx/vector_io/test_vector_stores.py::TestOgxVectorStores::test_vector_stores_search[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS]  TEST: TestOgxVectorStores.test_vector_stores_file_upload[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] [setup] STATUS: ERROR 2026-07-05T07:32:13.522581Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1110, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 728, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 908, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/ogx/conftest.py", line 292, in ogx_client wait_for_ogx_client_ready(client=client) ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 280, in wrapper for sample in TimeoutSampler( ~~~~~~~~~~~~~~^ func=func, ^^^^^^^^^^ ...<7 lines>... **kwargs, ^^^^^^^^^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 90 Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': } Last exception: N/A. [conftest] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  test_vector_stores_search[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS]  -------------------------------------------------------- SETUP -------------------------------------------------------- ERROR _ ERROR at setup of TestOgxVectorStores.test_vector_stores_search[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1091, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 908, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/ogx/conftest.py", line 292, in ogx_client wait_for_ogx_client_ready(client=client) ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 280, in wrapper for sample in TimeoutSampler( ~~~~~~~~~~~~~~^ func=func, ^^^^^^^^^^ ...<7 lines>... **kwargs, ^^^^^^^^^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 90 Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': } Last exception: N/A. _ 3 of 4 completed, 1 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 2 Error, 0 ReRun _ tests/ogx/vector_io/test_vector_stores.py::TestOgxVectorStores::test_response_file_search_tool_invocation[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS]  TEST: TestOgxVectorStores.test_vector_stores_search[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] [setup] STATUS: ERROR 2026-07-05T07:32:13.530975Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1091, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 908, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/ogx/conftest.py", line 292, in ogx_client wait_for_ogx_client_ready(client=client) ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 280, in wrapper for sample in TimeoutSampler( ~~~~~~~~~~~~~~^ func=func, ^^^^^^^^^^ ...<7 lines>... **kwargs, ^^^^^^^^^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 90 Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': } Last exception: N/A. [conftest] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  test_response_file_search_tool_invocation[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS]  -------------------------------------------------------- SETUP -------------------------------------------------------- ERROR TEST: TestOgxVectorStores.test_response_file_search_tool_invocation[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] [setup] STATUS: ERROR 2026-07-05T07:32:13.535158Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1091, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 908, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/ogx/conftest.py", line 292, in ogx_client wait_for_ogx_client_ready(client=client) ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 280, in wrapper for sample in TimeoutSampler( ~~~~~~~~~~~~~~^ func=func, ^^^^^^^^^^ ...<7 lines>... **kwargs, ^^^^^^^^^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 90 Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': } Last exception: N/A. [conftest] ------------------------------------------------------- TEARDOWN ------------------------------------------------------- 2026-07-05T07:32:13.535433Z [info ] Delete Route ogx-7168f58760ec  [ocp_resources Route] 2026-07-05T07:32:13.619523Z [info ] Deleting {'kind': 'Route', 'apiVersion': 'route.openshift.io/v1', 'metadata': {'name': 'ogx-7168f58760ec', 'namespace': 'test-ogx-vector-stores-f99f', 'uid': '57948085-ba79-4bc8-b9c3-afd2d2ec65c1', 'resourceVersion': '241453174', 'creationTimestamp': '2026-07-05T07:30:36Z', 'annotations': {'haproxy.router.openshift.io/timeout': '10m', 'openshift.io/host.generated': 'true'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'route.openshift.io/v1', 'time': '2026-07-05T07:30:36Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:haproxy.router.openshift.io/timeout': {}}}, 'f:spec': {'f:tls': {'.': {}, 'f:insecureEdgeTerminationPolicy': {}, 'f:termination': {}}, 'f:to': {'f:kind': {}, 'f:name': {}, 'f:weight': {}}, 'f:wildcardPolicy': {}}}}, {'manager': 'openshift-router', 'operation': 'Update', 'apiVersion': 'route.openshift.io/v1', 'time': '2026-07-05T07:30:36Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'f:ingress': {}}}, 'subresource': 'status'}]}, 'spec': {'host': 'ogx-7168f58760ec-test-ogx-vector-stores-f99f.apps.ods-qe-psi-23.osp.rh-ods.com', 'to': {'kind': 'Service', 'name': 'ogx-server-7f26f05f-service', 'weight': 100}, 'tls': {'termination': 'edge', 'insecureEdgeTerminationPolicy': 'Redirect'}, 'wildcardPolicy': 'None'}, 'status': {'ingress': [{'host': 'ogx-7168f58760ec-test-ogx-vector-stores-f99f.apps.ods-qe-psi-23.osp.rh-ods.com', 'routerName': 'default', 'conditions': [{'type': 'Admitted', 'status': 'True', 'lastTransitionTime': '2026-07-05T07:30:36Z'}], 'wildcardPolicy': 'None', 'routerCanonicalHostname': 'router-default.apps.ods-qe-psi-23.osp.rh-ods.com'}]}} [ocp_resources Route] 2026-07-05T07:32:13.635830Z [info ] Wait until Route ogx-7168f58760ec is deleted [ocp_resources Route] 2026-07-05T07:32:13.636007Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:32:13.649803Z [info ] Elapsed time: 6.794929504394531e-05 [0:00:00.000068] [timeout_sampler] 2026-07-05T07:32:13.649983Z [info ] Delete OGXServer ogx-server-7f26f05f [ocp_resources OGXServer] 2026-07-05T07:32:13.682804Z [info ] Deleting {'apiVersion': 'ogx.io/v1beta1', 'kind': 'OGXServer', 'metadata': {'creationTimestamp': '2026-07-05T07:30:05Z', 'generation': 1, 'managedFields': [{'apiVersion': 'ogx.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'.': {}, 'f:distribution': {'.': {}, 'f:name': {}}, 'f:network': {'.': {}, 'f:policy': {'.': {}, 'f:enabled': {}, 'f:ingress': {}}, 'f:port': {}}, 'f:workload': {'.': {}, 'f:overrides': {'.': {}, 'f:env': {}}, 'f:replicas': {}, 'f:resources': {'.': {}, 'f:limits': {'.': {}, 'f:cpu': {}, 'f:memory': {}}, 'f:requests': {'.': {}, 'f:cpu': {}, 'f:memory': {}}}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-07-05T07:30:05Z'}, {'apiVersion': 'ogx.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:availableReplicas': {}, 'f:conditions': {}, 'f:distributionConfig': {'.': {}, 'f:activeDistribution': {}, 'f:availableDistributions': {'.': {}, 'f:rh-dev': {}}}, 'f:phase': {}, 'f:serviceURL': {}, 'f:version': {'.': {}, 'f:lastUpdated': {}, 'f:operatorVersion': {}}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-07-05T07:32:07Z'}], 'name': 'ogx-server-7f26f05f', 'namespace': 'test-ogx-vector-stores-f99f', 'resourceVersion': '241455587', 'uid': '43b4dd94-255d-4855-871b-18109b215dc0'}, 'spec': {'distribution': {'name': 'rh-dev'}, 'network': {'policy': {'enabled': True, 'ingress': [{'from': [{'namespaceSelector': {'matchLabels': {'kubernetes.io/metadata.name': 'openshift-ingress'}}}], 'ports': [{'port': 8321, 'protocol': 'TCP'}]}]}, 'port': 8321}, 'workload': {'overrides': {'env': [{'name': 'INFERENCE_MODEL', 'value': 'Qwen3.6-35B-A3B'}, {'name': 'VLLM_API_TOKEN', 'valueFrom': {'secretKeyRef': {'key': 'vllm-api-token', 'name': 'ogx-distribution-secret'}}}, {'name': 'VLLM_URL', 'value': 'https://litellm-litemaas.apps.prod.rhoai.rh-aiservices-bu.com/v1'}, {'name': 'VLLM_TLS_VERIFY', 'value': 'false'}, {'name': 'VLLM_MAX_TOKENS', 'value': '80000'}, {'name': 'EMBEDDING_MODEL', 'value': 'Nomic-embed-text-v2-moe'}, {'name': 'EMBEDDING_PROVIDER_MODEL_ID', 'value': 'Nomic-embed-text-v2-moe'}, {'name': 'VLLM_EMBEDDING_URL', 'value': 'https://litellm-litemaas.apps.prod.rhoai.rh-aiservices-bu.com/v1'}, {'name': 'VLLM_EMBEDDING_API_TOKEN', 'valueFrom': {'secretKeyRef': {'key': 'vllm-embedding-api-token', 'name': 'ogx-distribution-secret'}}}, {'name': 'VLLM_EMBEDDING_MAX_TOKENS', 'value': '1024'}, {'name': 'VLLM_EMBEDDING_TLS_VERIFY', 'value': 'false'}, {'name': 'POSTGRES_HOST', 'value': 'vector-io-postgres-service'}, {'name': 'POSTGRES_PORT', 'value': '5432'}, {'name': 'POSTGRES_USER', 'valueFrom': {'secretKeyRef': {'key': 'postgres-user', 'name': 'ogx-distribution-secret'}}}, {'name': 'POSTGRES_PASSWORD', 'valueFrom': {'secretKeyRef': {'key': 'postgres-password', 'name': 'ogx-distribution-secret'}}}, {'name': 'POSTGRES_DB', 'value': 'ps_db'}, {'name': 'POSTGRES_TABLE_NAME', 'value': 'ogx_kvstore'}, {'name': 'ENABLE_S3', 'value': 's3'}, {'name': 'S3_BUCKET_NAME', 'value': 'opendatahub-tests'}, {'name': 'AWS_DEFAULT_REGION', 'value': 'us-east-1'}, {'name': 'S3_ENDPOINT_URL', 'value': 'https://s3.us-east-1.amazonaws.com/'}, {'name': 'AWS_ACCESS_KEY_ID', 'valueFrom': {'secretKeyRef': {'key': 'aws-access-key-id', 'name': 'ogx-distribution-secret'}}}, {'name': 'AWS_SECRET_ACCESS_KEY', 'valueFrom': {'secretKeyRef': {'key': 'aws-secret-access-key', 'name': 'ogx-distribution-secret'}}}, {'name': 'S3_AUTO_CREATE_BUCKET', 'value': 'true'}, {'name': 'MILVUS_ENDPOINT', 'value': 'http://vector-io-milvus-service:19530'}, {'name': 'MILVUS_TOKEN', 'valueFrom': {'secretKeyRef': {'key': 'milvus-token', 'name': 'vector-io-secret'}}}, {'name': 'MILVUS_CONSISTENCY_LEVEL', 'value': 'Bounded'}]}, 'replicas': 1, 'resources': {'limits': {'cpu': '2', 'memory': '2Gi'}, 'requests': {'cpu': '1', 'memory': '1Gi'}}}}, 'status': {'availableReplicas': 1, 'conditions': [{'lastTransitionTime': '2026-07-05T07:31:57Z', 'message': 'Deployment is ready', 'reason': 'DeploymentReady', 'status': 'True', 'type': 'DeploymentReady'}, {'lastTransitionTime': '2026-07-05T07:31:57Z', 'message': 'Service is ready', 'reason': 'ServiceReady', 'status': 'True', 'type': 'ServiceReady'}, {'lastTransitionTime': '2026-07-05T07:32:07Z', 'message': 'Health check passed', 'reason': 'HealthCheckPassed', 'status': 'True', 'type': 'HealthCheck'}], 'distributionConfig': {'activeDistribution': 'rh-dev', 'availableDistributions': {'rh-dev': 'registry.redhat.io/rhoai/odh-ogx-core-rhel9@sha256:3940a6ce94fe250d5d45634a73673bb5aa1a650e164cfecec1ad695fdd279242'}}, 'phase': 'Ready', 'serviceURL': 'http://ogx-server-7f26f05f-service.test-ogx-vector-stores-f99f.svc.cluster.local:8321', 'version': {'lastUpdated': '2026-07-05T07:32:07Z', 'operatorVersion': '"0.10.0"'}}} [ocp_resources OGXServer] 2026-07-05T07:32:13.701163Z [info ] Wait until OGXServer ogx-server-7f26f05f is deleted [ocp_resources OGXServer] 2026-07-05T07:32:13.701321Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:32:13.716356Z [info ] Elapsed time: 7.152557373046875e-05 [0:00:00.000072] [timeout_sampler] 2026-07-05T07:32:13.716567Z [info ] Delete Service vector-io-milvus-service [ocp_resources Service] 2026-07-05T07:32:13.749806Z [info ] Deleting {'kind': 'Service', 'apiVersion': 'v1', 'metadata': {'name': 'vector-io-milvus-service', 'namespace': 'test-ogx-vector-stores-f99f', 'uid': 'da87c369-689e-4bd5-9efc-6a35c47a1a69', 'resourceVersion': '241452277', 'creationTimestamp': '2026-07-05T07:30:05Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:30:05Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:internalTrafficPolicy': {}, 'f:ports': {'.': {}, 'k:{"port":19530,"protocol":"TCP"}': {'.': {}, 'f:name': {}, 'f:port': {}, 'f:protocol': {}, 'f:targetPort': {}}}, 'f:selector': {}, 'f:sessionAffinity': {}, 'f:type': {}}}}]}, 'spec': {'ports': [{'name': 'grpc', 'protocol': 'TCP', 'port': 19530, 'targetPort': 19530}], 'selector': {'app': 'milvus-standalone'}, 'clusterIP': '172.30.248.118', 'clusterIPs': ['172.30.248.118'], 'type': 'ClusterIP', 'sessionAffinity': 'None', 'ipFamilies': ['IPv4'], 'ipFamilyPolicy': 'SingleStack', 'internalTrafficPolicy': 'Cluster'}, 'status': {'loadBalancer': {}}} [ocp_resources Service] 2026-07-05T07:32:13.786306Z [info ] Wait until Service vector-io-milvus-service is deleted [ocp_resources Service] 2026-07-05T07:32:13.786469Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:32:13.798361Z [info ] Elapsed time: 3.266334533691406e-05 [0:00:00.000033] [timeout_sampler] 2026-07-05T07:32:13.798513Z [info ] Delete Deployment vector-io-milvus-deployment [ocp_resources Deployment] 2026-07-05T07:32:13.825674Z [info ] Deleting {'kind': 'Deployment', 'apiVersion': 'apps/v1', 'metadata': {'name': 'vector-io-milvus-deployment', 'namespace': 'test-ogx-vector-stores-f99f', 'uid': '56d69b31-9064-42e4-8feb-4bffd63dca5a', 'resourceVersion': '241452235', 'generation': 1, 'creationTimestamp': '2026-07-05T07:29:58Z', 'annotations': {'deployment.kubernetes.io/revision': '1'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'apps/v1', 'time': '2026-07-05T07:29:58Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:minReadySeconds': {}, 'f:progressDeadlineSeconds': {}, 'f:replicas': {}, 'f:revisionHistoryLimit': {}, 'f:selector': {}, 'f:strategy': {'f:type': {}}, 'f:template': {'f:metadata': {'f:labels': {'.': {}, 'f:app': {}}}, 'f:spec': {'f:containers': {'k:{"name":"milvus-standalone"}': {'.': {}, 'f:args': {}, 'f:env': {'.': {}, 'k:{"name":"COMMON_STORAGETYPE"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"DEPLOY_MODE"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"ETCD_ENDPOINTS"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"MINIO_ADDRESS"}': {'.': {}, 'f:name': {}}}, 'f:image': {}, 'f:imagePullPolicy': {}, 'f:name': {}, 'f:ports': {'.': {}, 'k:{"containerPort":19530,"protocol":"TCP"}': {'.': {}, 'f:containerPort': {}, 'f:protocol': {}}}, 'f:resources': {}, 'f:terminationMessagePath': {}, 'f:terminationMessagePolicy': {}, 'f:volumeMounts': {'.': {}, 'k:{"mountPath":"/var/lib/milvus"}': {'.': {}, 'f:mountPath': {}, 'f:name': {}}}}}, 'f:dnsPolicy': {}, 'f:restartPolicy': {}, 'f:schedulerName': {}, 'f:securityContext': {}, 'f:terminationGracePeriodSeconds': {}, 'f:volumes': {'.': {}, 'k:{"name":"milvus-data"}': {'.': {}, 'f:emptyDir': {}, 'f:name': {}}}}}}}}, {'manager': 'kube-controller-manager', 'operation': 'Update', 'apiVersion': 'apps/v1', 'time': '2026-07-05T07:30:04Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:deployment.kubernetes.io/revision': {}}}, 'f:status': {'f:availableReplicas': {}, 'f:conditions': {'.': {}, 'k:{"type":"Available"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:lastUpdateTime': {}, 'f:message': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"Progressing"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:lastUpdateTime': {}, 'f:message': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}}, 'f:observedGeneration': {}, 'f:readyReplicas': {}, 'f:replicas': {}, 'f:updatedReplicas': {}}}, 'subresource': 'status'}]}, 'spec': {'replicas': 1, 'selector': {'matchLabels': {'app': 'milvus-standalone'}}, 'template': {'metadata': {'creationTimestamp': None, 'labels': {'app': 'milvus-standalone'}}, 'spec': {'volumes': [{'name': 'milvus-data', 'emptyDir': {}}], 'containers': [{'name': 'milvus-standalone', 'image': 'quay.io/jgarciao/milvus@sha256:b7ae4fa7475f810837c91b61eb5aa39fd89f22348823fc76d9ce776a6ddc5844', 'args': ['milvus', 'run', 'standalone'], 'ports': [{'containerPort': 19530, 'protocol': 'TCP'}], 'env': [{'name': 'DEPLOY_MODE', 'value': 'standalone'}, {'name': 'ETCD_ENDPOINTS', 'value': 'vector-io-etcd-service:2379'}, {'name': 'MINIO_ADDRESS'}, {'name': 'COMMON_STORAGETYPE', 'value': 'local'}], 'resources': {}, 'volumeMounts': [{'name': 'milvus-data', 'mountPath': '/var/lib/milvus'}], 'terminationMessagePath': '/dev/termination-log', 'terminationMessagePolicy': 'File', 'imagePullPolicy': 'IfNotPresent'}], 'restartPolicy': 'Always', 'terminationGracePeriodSeconds': 30, 'dnsPolicy': 'ClusterFirst', 'securityContext': {}, 'schedulerName': 'default-scheduler'}}, 'strategy': {'type': 'Recreate'}, 'minReadySeconds': 5, 'revisionHistoryLimit': 10, 'progressDeadlineSeconds': 600}, 'status': {'observedGeneration': 1, 'replicas': 1, 'updatedReplicas': 1, 'readyReplicas': 1, 'availableReplicas': 1, 'conditions': [{'type': 'Available', 'status': 'True', 'lastUpdateTime': '2026-07-05T07:30:04Z', 'lastTransitionTime': '2026-07-05T07:30:04Z', 'reason': 'MinimumReplicasAvailable', 'message': 'Deployment has minimum availability.'}, {'type': 'Progressing', 'status': 'True', 'lastUpdateTime': '2026-07-05T07:30:04Z', 'lastTransitionTime': '2026-07-05T07:29:58Z', 'reason': 'NewReplicaSetAvailable', 'message': 'ReplicaSet "vector-io-milvus-deployment-6c56bf5769" has successfully progressed.'}]}} [ocp_resources Deployment] 2026-07-05T07:32:13.842864Z [info ] Wait until Deployment vector-io-milvus-deployment is deleted [ocp_resources Deployment] 2026-07-05T07:32:13.843020Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:32:13.855330Z [info ] Elapsed time: 7.534027099609375e-05 [0:00:00.000075] [timeout_sampler] 2026-07-05T07:32:13.855496Z [info ] Delete Secret vector-io-secret [ocp_resources Secret] 2026-07-05T07:32:13.880915Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'vector-io-secret', 'namespace': 'test-ogx-vector-stores-f99f', 'uid': 'a06ee82c-0ea2-4127-b97e-e9d144e3f8ba', 'resourceVersion': '241451996', 'creationTimestamp': '2026-07-05T07:29:58Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:29:58Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:milvus-token': {}, 'f:pgvector-password': {}, 'f:pgvector-user': {}, 'f:qdrant-api-key': {}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-05T07:32:13.896259Z [info ] Wait until Secret vector-io-secret is deleted [ocp_resources Secret] 2026-07-05T07:32:13.896424Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:32:13.908347Z [info ] Elapsed time: 0.00011563301086425781 [0:00:00.000116] [timeout_sampler] 2026-07-05T07:32:13.908566Z [info ] Delete Service vector-io-etcd-service [ocp_resources Service] 2026-07-05T07:32:13.932493Z [info ] Deleting {'kind': 'Service', 'apiVersion': 'v1', 'metadata': {'name': 'vector-io-etcd-service', 'namespace': 'test-ogx-vector-stores-f99f', 'uid': '75c25397-5b7c-4e13-a192-974c02aef4ec', 'resourceVersion': '241451990', 'creationTimestamp': '2026-07-05T07:29:57Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:29:57Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:internalTrafficPolicy': {}, 'f:ports': {'.': {}, 'k:{"port":2379,"protocol":"TCP"}': {'.': {}, 'f:port': {}, 'f:protocol': {}, 'f:targetPort': {}}}, 'f:selector': {}, 'f:sessionAffinity': {}, 'f:type': {}}}}]}, 'spec': {'ports': [{'protocol': 'TCP', 'port': 2379, 'targetPort': 2379}], 'selector': {'app': 'etcd'}, 'clusterIP': '172.30.151.126', 'clusterIPs': ['172.30.151.126'], 'type': 'ClusterIP', 'sessionAffinity': 'None', 'ipFamilies': ['IPv4'], 'ipFamilyPolicy': 'SingleStack', 'internalTrafficPolicy': 'Cluster'}, 'status': {'loadBalancer': {}}} [ocp_resources Service] 2026-07-05T07:32:13.963155Z [info ] Wait until Service vector-io-etcd-service is deleted [ocp_resources Service] 2026-07-05T07:32:13.963318Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:32:13.975092Z [info ] Elapsed time: 7.367134094238281e-05 [0:00:00.000074] [timeout_sampler] 2026-07-05T07:32:13.975302Z [info ] Delete Deployment vector-io-etcd-deployment [ocp_resources Deployment] 2026-07-05T07:32:14.004073Z [info ] Deleting {'kind': 'Deployment', 'apiVersion': 'apps/v1', 'metadata': {'name': 'vector-io-etcd-deployment', 'namespace': 'test-ogx-vector-stores-f99f', 'uid': 'f12c92f6-a80f-4045-8a05-e41c00cd054b', 'resourceVersion': '241451972', 'generation': 1, 'creationTimestamp': '2026-07-05T07:29:55Z', 'annotations': {'deployment.kubernetes.io/revision': '1'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'apps/v1', 'time': '2026-07-05T07:29:55Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:progressDeadlineSeconds': {}, 'f:replicas': {}, 'f:revisionHistoryLimit': {}, 'f:selector': {}, 'f:strategy': {'f:type': {}}, 'f:template': {'f:metadata': {'f:labels': {'.': {}, 'f:app': {}}}, 'f:spec': {'f:containers': {'k:{"name":"etcd"}': {'.': {}, 'f:command': {}, 'f:env': {'.': {}, 'k:{"name":"ETCD_AUTO_COMPACTION_MODE"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"ETCD_AUTO_COMPACTION_RETENTION"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"ETCD_QUOTA_BACKEND_BYTES"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"ETCD_SNAPSHOT_COUNT"}': {'.': {}, 'f:name': {}, 'f:value': {}}}, 'f:image': {}, 'f:imagePullPolicy': {}, 'f:name': {}, 'f:ports': {'.': {}, 'k:{"containerPort":2379,"protocol":"TCP"}': {'.': {}, 'f:containerPort': {}, 'f:protocol': {}}}, 'f:resources': {}, 'f:terminationMessagePath': {}, 'f:terminationMessagePolicy': {}, 'f:volumeMounts': {'.': {}, 'k:{"mountPath":"/etcd"}': {'.': {}, 'f:mountPath': {}, 'f:name': {}}}}}, 'f:dnsPolicy': {}, 'f:restartPolicy': {}, 'f:schedulerName': {}, 'f:securityContext': {}, 'f:terminationGracePeriodSeconds': {}, 'f:volumes': {'.': {}, 'k:{"name":"etcd-data"}': {'.': {}, 'f:emptyDir': {}, 'f:name': {}}}}}}}}, {'manager': 'kube-controller-manager', 'operation': 'Update', 'apiVersion': 'apps/v1', 'time': '2026-07-05T07:29:57Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:deployment.kubernetes.io/revision': {}}}, 'f:status': {'f:availableReplicas': {}, 'f:conditions': {'.': {}, 'k:{"type":"Available"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:lastUpdateTime': {}, 'f:message': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"Progressing"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:lastUpdateTime': {}, 'f:message': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}}, 'f:observedGeneration': {}, 'f:readyReplicas': {}, 'f:replicas': {}, 'f:updatedReplicas': {}}}, 'subresource': 'status'}]}, 'spec': {'replicas': 1, 'selector': {'matchLabels': {'app': 'etcd'}}, 'template': {'metadata': {'creationTimestamp': None, 'labels': {'app': 'etcd'}}, 'spec': {'volumes': [{'name': 'etcd-data', 'emptyDir': {}}], 'containers': [{'name': 'etcd', 'image': 'quay.io/coreos/etcd@sha256:3397341272b9e0a6f44d7e3fc7c321c6efe6cbe82ce866b9b01d0c704bfc5bf3', 'command': ['etcd', '--advertise-client-urls=http://vector-io-etcd-service:2379', '--listen-client-urls=http://0.0.0.0:2379', '--data-dir=/etcd'], 'ports': [{'containerPort': 2379, 'protocol': 'TCP'}], 'env': [{'name': 'ETCD_AUTO_COMPACTION_MODE', 'value': 'revision'}, {'name': 'ETCD_AUTO_COMPACTION_RETENTION', 'value': '1000'}, {'name': 'ETCD_QUOTA_BACKEND_BYTES', 'value': '4294967296'}, {'name': 'ETCD_SNAPSHOT_COUNT', 'value': '50000'}], 'resources': {}, 'volumeMounts': [{'name': 'etcd-data', 'mountPath': '/etcd'}], 'terminationMessagePath': '/dev/termination-log', 'terminationMessagePolicy': 'File', 'imagePullPolicy': 'IfNotPresent'}], 'restartPolicy': 'Always', 'terminationGracePeriodSeconds': 30, 'dnsPolicy': 'ClusterFirst', 'securityContext': {}, 'schedulerName': 'default-scheduler'}}, 'strategy': {'type': 'Recreate'}, 'revisionHistoryLimit': 10, 'progressDeadlineSeconds': 600}, 'status': {'observedGeneration': 1, 'replicas': 1, 'updatedReplicas': 1, 'readyReplicas': 1, 'availableReplicas': 1, 'conditions': [{'type': 'Available', 'status': 'True', 'lastUpdateTime': '2026-07-05T07:29:57Z', 'lastTransitionTime': '2026-07-05T07:29:57Z', 'reason': 'MinimumReplicasAvailable', 'message': 'Deployment has minimum availability.'}, {'type': 'Progressing', 'status': 'True', 'lastUpdateTime': '2026-07-05T07:29:57Z', 'lastTransitionTime': '2026-07-05T07:29:55Z', 'reason': 'NewReplicaSetAvailable', 'message': 'ReplicaSet "vector-io-etcd-deployment-5784dbfbb6" has successfully progressed.'}]}} [ocp_resources Deployment] 2026-07-05T07:32:14.018102Z [info ] Wait until Deployment vector-io-etcd-deployment is deleted [ocp_resources Deployment] 2026-07-05T07:32:14.018250Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:32:14.032054Z [info ] Elapsed time: 7.605552673339844e-05 [0:00:00.000076] [timeout_sampler] 2026-07-05T07:32:14.032197Z [info ] Delete Service vector-io-postgres-service [ocp_resources Service] 2026-07-05T07:32:14.055676Z [info ] Deleting {'kind': 'Service', 'apiVersion': 'v1', 'metadata': {'name': 'vector-io-postgres-service', 'namespace': 'test-ogx-vector-stores-f99f', 'uid': 'b5d2534e-2448-466a-9e43-b8827250e543', 'resourceVersion': '241451903', 'creationTimestamp': '2026-07-05T07:29:55Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:29:55Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:internalTrafficPolicy': {}, 'f:ports': {'.': {}, 'k:{"port":5432,"protocol":"TCP"}': {'.': {}, 'f:port': {}, 'f:protocol': {}, 'f:targetPort': {}}}, 'f:selector': {}, 'f:sessionAffinity': {}, 'f:type': {}}}}]}, 'spec': {'ports': [{'protocol': 'TCP', 'port': 5432, 'targetPort': 5432}], 'selector': {'app': 'postgres'}, 'clusterIP': '172.30.107.65', 'clusterIPs': ['172.30.107.65'], 'type': 'ClusterIP', 'sessionAffinity': 'None', 'ipFamilies': ['IPv4'], 'ipFamilyPolicy': 'SingleStack', 'internalTrafficPolicy': 'Cluster'}, 'status': {'loadBalancer': {}}} [ocp_resources Service] 2026-07-05T07:32:14.089248Z [info ] Wait until Service vector-io-postgres-service is deleted [ocp_resources Service] 2026-07-05T07:32:14.089404Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:32:14.101270Z [info ] Elapsed time: 7.653236389160156e-05 [0:00:00.000077] [timeout_sampler] 2026-07-05T07:32:14.101430Z [info ] Delete Deployment vector-io-postgres-deployment [ocp_resources Deployment] 2026-07-05T07:32:14.128654Z [info ] Deleting {'kind': 'Deployment', 'apiVersion': 'apps/v1', 'metadata': {'name': 'vector-io-postgres-deployment', 'namespace': 'test-ogx-vector-stores-f99f', 'uid': '0dc4910e-e11a-4f36-b7b9-282e58beca04', 'resourceVersion': '241451898', 'generation': 1, 'creationTimestamp': '2026-07-05T07:29:49Z', 'annotations': {'deployment.kubernetes.io/revision': '1'}, 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'apps/v1', 'time': '2026-07-05T07:29:49Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:minReadySeconds': {}, 'f:progressDeadlineSeconds': {}, 'f:replicas': {}, 'f:revisionHistoryLimit': {}, 'f:selector': {}, 'f:strategy': {'f:type': {}}, 'f:template': {'f:metadata': {'f:labels': {'.': {}, 'f:app': {}}}, 'f:spec': {'f:containers': {'k:{"name":"postgres"}': {'.': {}, 'f:env': {'.': {}, 'k:{"name":"POSTGRESQL_DATABASE"}': {'.': {}, 'f:name': {}, 'f:value': {}}, 'k:{"name":"POSTGRESQL_PASSWORD"}': {'.': {}, 'f:name': {}, 'f:valueFrom': {'.': {}, 'f:secretKeyRef': {}}}, 'k:{"name":"POSTGRESQL_USER"}': {'.': {}, 'f:name': {}, 'f:valueFrom': {'.': {}, 'f:secretKeyRef': {}}}}, 'f:image': {}, 'f:imagePullPolicy': {}, 'f:name': {}, 'f:ports': {'.': {}, 'k:{"containerPort":5432,"protocol":"TCP"}': {'.': {}, 'f:containerPort': {}, 'f:protocol': {}}}, 'f:resources': {}, 'f:terminationMessagePath': {}, 'f:terminationMessagePolicy': {}, 'f:volumeMounts': {'.': {}, 'k:{"mountPath":"/var/lib/pgsql/data"}': {'.': {}, 'f:mountPath': {}, 'f:name': {}}}}}, 'f:dnsPolicy': {}, 'f:restartPolicy': {}, 'f:schedulerName': {}, 'f:securityContext': {}, 'f:terminationGracePeriodSeconds': {}, 'f:volumes': {'.': {}, 'k:{"name":"postgresdata"}': {'.': {}, 'f:emptyDir': {}, 'f:name': {}}}}}}}}, {'manager': 'kube-controller-manager', 'operation': 'Update', 'apiVersion': 'apps/v1', 'time': '2026-07-05T07:29:55Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:deployment.kubernetes.io/revision': {}}}, 'f:status': {'f:availableReplicas': {}, 'f:conditions': {'.': {}, 'k:{"type":"Available"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:lastUpdateTime': {}, 'f:message': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"Progressing"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:lastUpdateTime': {}, 'f:message': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}}, 'f:observedGeneration': {}, 'f:readyReplicas': {}, 'f:replicas': {}, 'f:updatedReplicas': {}}}, 'subresource': 'status'}]}, 'spec': {'replicas': 1, 'selector': {'matchLabels': {'app': 'postgres'}}, 'template': {'metadata': {'creationTimestamp': None, 'labels': {'app': 'postgres'}}, 'spec': {'volumes': [{'name': 'postgresdata', 'emptyDir': {}}], 'containers': [{'name': 'postgres', 'image': 'registry.redhat.io/rhel9/postgresql-15@sha256:90ec347a35ab8a5d530c8d09f5347b13cc71df04f3b994bfa8b1a409b1171d59', 'ports': [{'containerPort': 5432, 'protocol': 'TCP'}], 'env': [{'name': 'POSTGRESQL_DATABASE', 'value': 'ps_db'}, {'name': 'POSTGRESQL_USER', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'postgres-user'}}}, {'name': 'POSTGRESQL_PASSWORD', 'valueFrom': {'secretKeyRef': {'name': 'ogx-distribution-secret', 'key': 'postgres-password'}}}], 'resources': {}, 'volumeMounts': [{'name': 'postgresdata', 'mountPath': '/var/lib/pgsql/data'}], 'terminationMessagePath': '/dev/termination-log', 'terminationMessagePolicy': 'File', 'imagePullPolicy': 'IfNotPresent'}], 'restartPolicy': 'Always', 'terminationGracePeriodSeconds': 30, 'dnsPolicy': 'ClusterFirst', 'securityContext': {}, 'schedulerName': 'default-scheduler'}}, 'strategy': {'type': 'Recreate'}, 'minReadySeconds': 5, 'revisionHistoryLimit': 10, 'progressDeadlineSeconds': 600}, 'status': {'observedGeneration': 1, 'replicas': 1, 'updatedReplicas': 1, 'readyReplicas': 1, 'availableReplicas': 1, 'conditions': [{'type': 'Available', 'status': 'True', 'lastUpdateTime': '2026-07-05T07:29:55Z', 'lastTransitionTime': '2026-07-05T07:29:55Z', 'reason': 'MinimumReplicasAvailable', 'message': 'Deployment has minimum availability.'}, {'type': 'Progressing', 'status': 'True', 'lastUpdateTime': '2026-07-05T07:29:55Z', 'lastTransitionTime': '2026-07-05T07:29:49Z', 'reason': 'NewReplicaSetAvailable', 'message': 'ReplicaSet "vector-io-postgres-deployment-5db8dfbb79" has successfully progressed.'}]}} [ocp_resources Deployment] 2026-07-05T07:32:14.143965Z [info ] Wait until Deployment vector-io-postgres-deployment is deleted [ocp_resources Deployment] 2026-07-05T07:32:14.144112Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:32:14.156380Z [info ] Elapsed time: 7.390975952148438e-05 [0:00:00.000074] [timeout_sampler] 2026-07-05T07:32:14.156587Z [info ] Delete Secret ogx-distribution-secret [ocp_resources Secret] 2026-07-05T07:32:14.182039Z [info ] Deleting {'kind': 'Secret', 'apiVersion': 'v1', 'metadata': {'name': 'ogx-distribution-secret', 'namespace': 'test-ogx-vector-stores-f99f', 'uid': 'd8353096-4165-485f-aae3-71863b2903db', 'resourceVersion': '241451705', 'creationTimestamp': '2026-07-05T07:29:49Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:29:49Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:aws-access-key-id': {}, 'f:aws-secret-access-key': {}, 'f:postgres-password': {}, 'f:postgres-user': {}, 'f:vllm-api-token': {}, 'f:vllm-embedding-api-token': {}}, 'f:type': {}}}]}, 'data': '*******', 'type': 'Opaque'} [ocp_resources Secret] 2026-07-05T07:32:14.198029Z [info ] Wait until Secret ogx-distribution-secret is deleted [ocp_resources Secret] 2026-07-05T07:32:14.198148Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:32:14.210520Z [info ] Elapsed time: 9.608268737792969e-05 [0:00:00.000096] [timeout_sampler] 2026-07-05T07:32:14.331571Z [info ] Delete Project test-ogx-vector-stores-f99f [ocp_resources Project] 2026-07-05T07:32:14.363040Z [info ] Deleting {'kind': 'Project', 'apiVersion': 'project.openshift.io/v1', 'metadata': {'name': 'test-ogx-vector-stores-f99f', 'uid': 'f2b72a89-08c7-421c-b7f2-a06a57bf9405', 'resourceVersion': '241455750', 'creationTimestamp': '2026-07-05T07:29:49Z', 'labels': {'kubernetes.io/metadata.name': 'test-ogx-vector-stores-f99f', 'pod-security.kubernetes.io/audit': 'restricted', 'pod-security.kubernetes.io/audit-version': 'latest', 'pod-security.kubernetes.io/enforce': 'restricted', 'pod-security.kubernetes.io/enforce-version': 'latest', 'pod-security.kubernetes.io/warn': 'restricted', 'pod-security.kubernetes.io/warn-version': 'latest'}, 'annotations': {'openshift.io/description': '', 'openshift.io/display-name': '', 'openshift.io/requester': 'system:serviceaccount:kube-system:olminstall-cluster-admin', 'openshift.io/sa.scc.mcs': 's0:c31,c30', 'openshift.io/sa.scc.supplemental-groups': '1000990000/10000', 'openshift.io/sa.scc.uid-range': '1000990000/10000', 'security.openshift.io/MinimallySufficientPodSecurityStandard': 'restricted'}, 'managedFields': [{'manager': 'pod-security-admission-label-synchronization-controller', 'operation': 'Apply', 'apiVersion': 'v1', 'time': '2026-07-05T07:32:13Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:security.openshift.io/MinimallySufficientPodSecurityStandard': {}}, 'f:labels': {'f:pod-security.kubernetes.io/audit': {}, 'f:pod-security.kubernetes.io/audit-version': {}, 'f:pod-security.kubernetes.io/enforce': {}, 'f:pod-security.kubernetes.io/enforce-version': {}, 'f:pod-security.kubernetes.io/warn': {}, 'f:pod-security.kubernetes.io/warn-version': {}}}}}, {'manager': 'cluster-policy-controller', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:29:49Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:openshift.io/sa.scc.mcs': {}, 'f:openshift.io/sa.scc.supplemental-groups': {}, 'f:openshift.io/sa.scc.uid-range': {}}}}}, {'manager': 'openshift-apiserver', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:29:49Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:openshift.io/description': {}, 'f:openshift.io/display-name': {}, 'f:openshift.io/requester': {}}, 'f:labels': {'.': {}, 'f:kubernetes.io/metadata.name': {}}}}}, {'manager': 'openshift-controller-manager', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-07-05T07:29:49Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:finalizers': {}}}, 'subresource': 'finalize'}]}, 'spec': {'finalizers': ['kubernetes']}, 'status': {'phase': 'Active'}} [ocp_resources Project] 2026-07-05T07:32:14.383142Z [info ] Wait until Project test-ogx-vector-stores-f99f is deleted [ocp_resources Project] 2026-07-05T07:32:14.383295Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-07-05T07:33:00.231130Z [info ] Elapsed time: 45.831383228302 [0:00:45.831383] [timeout_sampler] _ ERROR at setup of TestOgxVectorStores.test_response_file_search_tool_invocation[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1091, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 908, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/ogx/conftest.py", line 292, in ogx_client wait_for_ogx_client_ready(client=client) ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 280, in wrapper for sample in TimeoutSampler( ~~~~~~~~~~~~~~^ func=func, ^^^^^^^^^^ ...<7 lines>... **kwargs, ^^^^^^^^^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 90 Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': } Last exception: N/A. _ 4 of 4 completed, 1 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 3 Error, 0 ReRun _ ===== 1 passed, 31 deselected, 199 warnings, 3 errors in 386.40s (0:06:26) ===== ==================================== ERRORS ==================================== _ ERROR at setup of TestOgxVectorStores.test_vector_stores_file_upload[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1110, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 728, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 908, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/ogx/conftest.py", line 292, in ogx_client wait_for_ogx_client_ready(client=client) ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 280, in wrapper for sample in TimeoutSampler( ~~~~~~~~~~~~~~^ func=func, ^^^^^^^^^^ ...<7 lines>... **kwargs, ^^^^^^^^^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 90 Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': } Last exception: N/A. _ ERROR at setup of TestOgxVectorStores.test_vector_stores_search[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1091, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 908, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/ogx/conftest.py", line 292, in ogx_client wait_for_ogx_client_ready(client=client) ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 280, in wrapper for sample in TimeoutSampler( ~~~~~~~~~~~~~~^ func=func, ^^^^^^^^^^ ...<7 lines>... **kwargs, ^^^^^^^^^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 90 Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': } Last exception: N/A. _ ERROR at setup of TestOgxVectorStores.test_response_file_search_tool_invocation[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1091, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 908, in call_fixture_func fixture_result = next(generator) File "/home/odh/opendatahub-tests/tests/ogx/conftest.py", line 292, in ogx_client wait_for_ogx_client_ready(client=client) ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 280, in wrapper for sample in TimeoutSampler( ~~~~~~~~~~~~~~^ func=func, ^^^^^^^^^^ ...<7 lines>... **kwargs, ^^^^^^^^^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 182, in __iter__ raise TimeoutExpiredError(self._get_exception_log(exp=last_exp), last_exp=last_exp) timeout_sampler.TimeoutExpiredError: Timed Out: 90 Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': } Last exception: N/A. =============================== warnings summary =============================== tests/ogx/vector_io/test_vector_stores_evaluation.py:6 tests/ogx/vector_io/test_vector_stores_evaluation.py:6 /home/odh/opendatahub-tests/tests/ogx/vector_io/test_vector_stores_evaluation.py:6: DeprecationWarning: Importing AnswerRelevancy from 'ragas.metrics' is deprecated and will be removed in v1.0. Please use 'ragas.metrics.collections' instead. Example: from ragas.metrics.collections import AnswerRelevancy from ragas.metrics import AnswerRelevancy, ContextPrecision, ContextRecall, Faithfulness tests/ogx/vector_io/test_vector_stores_evaluation.py:6 tests/ogx/vector_io/test_vector_stores_evaluation.py:6 /home/odh/opendatahub-tests/tests/ogx/vector_io/test_vector_stores_evaluation.py:6: DeprecationWarning: Importing ContextPrecision from 'ragas.metrics' is deprecated and will be removed in v1.0. Please use 'ragas.metrics.collections' instead. Example: from ragas.metrics.collections import ContextPrecision from ragas.metrics import AnswerRelevancy, ContextPrecision, ContextRecall, Faithfulness tests/ogx/vector_io/test_vector_stores_evaluation.py:6 tests/ogx/vector_io/test_vector_stores_evaluation.py:6 /home/odh/opendatahub-tests/tests/ogx/vector_io/test_vector_stores_evaluation.py:6: DeprecationWarning: Importing ContextRecall from 'ragas.metrics' is deprecated and will be removed in v1.0. Please use 'ragas.metrics.collections' instead. Example: from ragas.metrics.collections import ContextRecall from ragas.metrics import AnswerRelevancy, ContextPrecision, ContextRecall, Faithfulness tests/ogx/vector_io/test_vector_stores_evaluation.py:6 tests/ogx/vector_io/test_vector_stores_evaluation.py:6 /home/odh/opendatahub-tests/tests/ogx/vector_io/test_vector_stores_evaluation.py:6: DeprecationWarning: Importing Faithfulness from 'ragas.metrics' is deprecated and will be removed in v1.0. Please use 'ragas.metrics.collections' instead. Example: from ragas.metrics.collections import Faithfulness from ragas.metrics import AnswerRelevancy, ContextPrecision, ContextRecall, Faithfulness tests/ogx/operator/test_ogx_server.py: 4 warnings tests/ogx/vector_io/test_vector_stores.py: 187 warnings /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1110: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: /workspace/tests-shared/tests-payload/results/ogx-smoke.xml - =========================== short test summary info ============================ ERROR tests/ogx/vector_io/test_vector_stores.py::TestOgxVectorStores::test_vector_stores_file_upload[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] - timeout_sampler.TimeoutExpiredError: Timed Out: 90 Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': } Last exception: N/A. ERROR tests/ogx/vector_io/test_vector_stores.py::TestOgxVectorStores::test_vector_stores_search[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] - timeout_sampler.TimeoutExpiredError: Timed Out: 90 Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': } Last exception: N/A. ERROR tests/ogx/vector_io/test_vector_stores.py::TestOgxVectorStores::test_response_file_search_tool_invocation[vector_io:milvus-remote, files: s3, embedding: vllm-embedding, dataset:IBM_2025_Q4_EARNINGS] - timeout_sampler.TimeoutExpiredError: Timed Out: 90 Function: tests.ogx.utils.wait_for_ogx_client_ready Kwargs: {'client': } Last exception: N/A. ===== 1 passed, 31 deselected, 199 warnings, 3 errors in 386.40s (0:06:26) =====