--- BVT pytest attempt 2026-06-11T09:08:45.940840+00:00 (/.local/bin/uv run pytest...) --- 2026-06-11T09:08:47.092385Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-06-11T09:08:47.101288Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.101841Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.102204Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.102372Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.102529Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.102670Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.102798Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.102913Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.103624Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-11T09:08:47.110565Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-06-11T09:08:47.210473Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-11T09:08:47.237719Z [info ] Running model registry tests against default namespace [conftest] 2026-06-11T09:08:47.238020Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] ============================= test session starts ============================== platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.14.5', 'Platform': 'Linux-5.14.0-427.115.1.el9_4.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.0.3', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.13.0', 'asyncio': '1.3.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.3.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.1.0'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.13.0, asyncio-1.3.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.3.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.1.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... 2026-06-11T09:08:47.847763Z [info ] Number of unique LMEval tasks with more than 10000 downloads: 35 [tests.ai_safety.lm_eval.utils] name=tests.ai_safety.lm_eval.utils 2026-06-11T09:08:47.869971Z [info ] Number of unique LMEval tasks with more than 1547.0 downloads: 43 [tests.ai_safety.lm_eval.utils] name=tests.ai_safety.lm_eval.utils collected 273 items / 268 deselected / 5 selected tests/ai_safety/evalhub/test_evalhub_deployment.py::test_evalhub_crd_exists  ----------------------------------------------- test_evalhub_crd_exists ----------------------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T09:08:47.976650Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() started [sqlalchemy.orm.mapper.Mapper] 2026-06-11T09:08:47.977802Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop id [sqlalchemy.orm.mapper.Mapper] 2026-06-11T09:08:47.977950Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop test_name [sqlalchemy.orm.mapper.Mapper] 2026-06-11T09:08:47.978084Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop start_time [sqlalchemy.orm.mapper.Mapper] 2026-06-11T09:08:47.978209Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() complete [sqlalchemy.orm.mapper.Mapper] 2026-06-11T09:08:47.978345Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.978771Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.979125Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.979276Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.979466Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/ai_safety/evalhub/test_evalhub_deployment.py::test_evalhub_crd_exists', 1781168927) [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.979610Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/ai_safety/evalhub/test_evalhub_deployment.py::test_evalhub_crd_exists', 1781168927) [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.979737Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.979859Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T09:08:47.981025Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-06-11T09:08:47.981168Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-06-11T09:08:47.981292Z [info ] Executing session fixture: admin_client [conftest] 2026-06-11T09:08:47.983305Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-11T09:08:47.988269Z [info ] Executing session fixture: dsc_resource [conftest] 2026-06-11T09:08:47.988600Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-06-11T09:08:48.062235Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-06-11T09:08:48.062387Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-06-11T09:08:48.063594Z [info ] Executing session fixture: os_path_environment [conftest] 2026-06-11T09:08:48.064028Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-06-11T09:08:48.064179Z [info ] Executing session fixture: bin_directory [conftest] 2026-06-11T09:08:48.065718Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-06-11T09:08:48.066001Z [info ] kind: ConsoleCLIDownload api version: console.openshift.io/v1 [ocp_resources.resource] 2026-06-11T09:08:48.089691Z [info ] All oc console cli download links: ['https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar'] [utilities.infra] name=utilities.infra 2026-06-11T09:08:48.089853Z [info ] Downloading archive using: url=https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar [utilities.infra] name=utilities.infra 2026-06-11T09:08:49.571529Z [info ] Extract the downloaded archive. [utilities.infra] name=utilities.infra 2026-06-11T09:08:49.857466Z [info ] Downloaded file: ['oc']  [utilities.infra] name=utilities.infra 2026-06-11T09:08:49.888676Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-06-11T09:08:49.886099Z [info ] OC binary path: /tmp/pytest-6yqWyeAJhTs334UPeoFRTk/bin0/oc [tests.conftest] name=tests.conftest 2026-06-11T09:08:49.886167Z [info ] Adding /tmp/pytest-6yqWyeAJhTs334UPeoFRTk/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-06-11T09:08:49.889310Z [info ] Executing session fixture: nodes [conftest] 2026-06-11T09:08:49.933239Z [info ] Executing session fixture: dsci_resource [conftest] 2026-06-11T09:08:49.933437Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-11T09:08:49.947067Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-06-11T09:08:49.947392Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-06-11T09:08:49.947557Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-06-11T09:08:49.946221Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-06-11T09:08:49.947784Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-06-11T09:08:50.031998Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-06-11T09:08:50.114800Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsci_status_ready Kwargs: {'dsci_resource': }) [timeout_sampler] 2026-06-11T09:08:50.113984Z [info ] Wait for DSCI default-dsci to be in Ready status. [utilities.infra] name=utilities.infra 2026-06-11T09:08:50.115278Z [info ] Get DSCInitialization default-dsci status [ocp_resources DSCInitialization] 2026-06-11T09:08:50.128072Z [info ] Elapsed time: 0.00011086463928222656 [0:00:00.000111] [timeout_sampler] 2026-06-11T09:08:50.128242Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': }) [timeout_sampler] 2026-06-11T09:08:50.127313Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:08:50.128535Z [info ] Get DataScienceCluster default-dsc status [ocp_resources DataScienceCluster] 2026-06-11T09:08:55.161578Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:09:00.204545Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:09:05.239716Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:09:10.301825Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:09:15.354884Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:09:20.409116Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:09:25.448673Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:09:30.487839Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:09:35.535578Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:09:40.589054Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:09:45.679325Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:09:50.734187Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:09:55.786940Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:10:00.833340Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:10:05.867391Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:10:10.909945Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:10:15.954754Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:10:20.999091Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:10:26.044727Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:10:31.083296Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:10:36.116671Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:10:41.175463Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:10:46.220807Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:10:51.345816Z [error ] Cluster sanity check failed: Timed Out: 120 Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': } Last exception: ResourceNotReadyError: DSC default-dsc is not ready. Current status: {'components': {'aipipelines': {'managementState': 'Removed'}, 'dashboard': {'managementState': 'Managed'}, 'feastoperator': {'managementState': 'Removed'}, 'kserve': {'managementState': 'Managed', 'releases': [{'name': 'KServe', 'repoUrl': 'https://github.com/kserve/kserve/', 'version': 'v0.17.0'}, {'name': 'vLLM', 'repoUrl': 'https://github.com/vllm-project/vllm', 'version': 'v0.18.0'}, {'name': 'llm-d-inference-scheduler', 'repoUrl': 'https://github.com/llm-d/llm-d-inference-scheduler', 'version': 'v0.7.1'}, {'name': 'llm-d-workload-variant-autoscaler', 'repoUrl': 'https://github.com/llm-d/llm-d-workload-variant-autoscaler', 'version': 'v0.6.0'}]}, 'kueue': {'managementState': 'Removed'}, 'llamastackoperator': {'managementState': 'Removed'}, 'mlflowoperator': {'managementState': 'Removed'}, 'modelregistry': {'managementState': 'Managed', 'registriesNamespace': 'rhoai-model-registries', 'releases': [{'name': 'Kubeflow Model Registry', 'repoUrl': 'https://github.com/kubeflow/model-registry', 'version': 'latest'}, {'name': 'Open Data Hub Model Registry Operator', 'repoUrl': 'https://github.com/opendatahub-io/model-registry-operator', 'version': 'latest'}]}, 'ray': {'managementState': 'Removed'}, 'sparkoperator': {'managementState': 'Removed'}, 'trainer': {'managementState': 'Removed'}, 'trainingoperator': {'managementState': 'Removed'}, 'trustyai': {'managementState': 'Managed', 'releases': [{'name': 'TrustyAI operator', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI service', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-explainability', 'version': 'v0.28.0'}, {'name': 'TrustyAI LMEval driver', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI LMEval job', 'repoUrl': 'https://github.com/EleutherAI/lm-evaluation-harness', 'version': 'v0.4.8'}, {'name': 'TrustyAI Guardrails orchestrator', 'repoUrl': 'https://github.com/foundation-model-stack/fms-guardrails-orchestrator', 'version': '0.9.4'}, {'name': 'TrustyAI builtin detectors', 'repoUrl': 'https://github.com/trustyai-explainability/guardrails-regex-detector', 'version': 'v0.2.0'}, {'name': 'TrustyAI sidecar gateway', 'repoUrl': 'https://github.com/trustyai-explainability/vllm-orchestrator-gateway', 'version': 'v0.2.1'}]}, 'workbenches': {'managementState': 'Managed', 'releases': [{'name': 'Kubeflow Notebook Controller', 'repoUrl': 'https://github.com/kubeflow/kubeflow', 'version': '1.10.0'}], 'workbenchNamespace': 'rhods-notebooks'}}, 'conditions': [{'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'reason': 'Error', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'observedGeneration': 3, 'reason': 'Error', 'status': 'False', 'type': 'ProvisioningSucceeded'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'ComponentsReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'AIPipelinesReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'DashboardReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'FeastOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceDependencies'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceWideEPDependencies'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'KserveReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'KueueReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'LlamaStackOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'MLflowOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'ModelControllerReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'ModelRegistryReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'ModelsAsServiceReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'RayReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'SparkOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainerReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainingOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'TrustyAIReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'WorkbenchesReady'}], 'observedGeneration': 1, 'phase': 'Not Ready', 'release': {'name': 'OpenShift AI Self-Managed', 'version': '3.4.1'}}. [utilities.infra] name=utilities.infra ================ 268 deselected, 1 warning in 124.78s (0:02:04) ================ ================ 268 deselected, 1 warning in 124.78s (0:02:04) ================ ! _pytest.outcomes.Exit: Cluster sanity check failed: Timed Out: 120 Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': } Last exception: ResourceNotReadyError: DSC default-dsc is not ready. Current status: {'components': {'aipipelines': {'managementState': 'Removed'}, 'dashboard': {'managementState': 'Managed'}, 'feastoperator': {'managementState': 'Removed'}, 'kserve': {'managementState': 'Managed', 'releases': [{'name': 'KServe', 'repoUrl': 'https://github.com/kserve/kserve/', 'version': 'v0.17.0'}, {'name': 'vLLM', 'repoUrl': 'https://github.com/vllm-project/vllm', 'version': 'v0.18.0'}, {'name': 'llm-d-inference-scheduler', 'repoUrl': 'https://github.com/llm-d/llm-d-inference-scheduler', 'version': 'v0.7.1'}, {'name': 'llm-d-workload-variant-autoscaler', 'repoUrl': 'https://github.com/llm-d/llm-d-workload-variant-autoscaler', 'version': 'v0.6.0'}]}, 'kueue': {'managementState': 'Removed'}, 'llamastackoperator': {'managementState': 'Removed'}, 'mlflowoperator': {'managementState': 'Removed'}, 'modelregistry': {'managementState': 'Managed', 'registriesNamespace': 'rhoai-model-registries', 'releases': [{'name': 'Kubeflow Model Registry', 'repoUrl': 'https://github.com/kubeflow/model-registry', 'version': 'latest'}, {'name': 'Open Data Hub Model Registry Operator', 'repoUrl': 'https://github.com/opendatahub-io/model-registry-operator', 'version': 'latest'}]}, 'ray': {'managementState': 'Removed'}, 'sparkoperator': {'managementState': 'Removed'}, 'trainer': {'managementState': 'Removed'}, 'trainingoperator': {'managementState': 'Removed'}, 'trustyai': {'managementState': 'Managed', 'releases': [{'name': 'TrustyAI operator', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI service', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-explainability', 'version': 'v0.28.0'}, {'name': 'TrustyAI LMEval driver', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI LMEval job', 'repoUrl': 'https://github.com/EleutherAI/lm-evaluation-harness', 'version': 'v0.4.8'}, {'name': 'TrustyAI Guardrails orchestrator', 'repoUrl': 'https://github.com/foundation-model-stack/fms-guardrails-orchestrator', 'version': '0.9.4'}, {'name': 'TrustyAI builtin detectors', 'repoUrl': 'https://github.com/trustyai-explainability/guardrails-regex-detector', 'version': 'v0.2.0'}, {'name': 'TrustyAI sidecar gateway', 'repoUrl': 'https://github.com/trustyai-explainability/vllm-orchestrator-gateway', 'version': 'v0.2.1'}]}, 'workbenches': {'managementState': 'Managed', 'releases': [{'name': 'Kubeflow Notebook Controller', 'repoUrl': 'https://github.com/kubeflow/kubeflow', 'version': '1.10.0'}], 'workbenchNamespace': 'rhods-notebooks'}}, 'conditions': [{'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'reason': 'Error', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'observedGeneration': 3, 'reason': 'Error', 'status': 'False', 'type': 'ProvisioningSucceeded'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'ComponentsReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'AIPipelinesReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'DashboardReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'FeastOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceDependencies'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceWideEPDependencies'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'KserveReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'KueueReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'LlamaStackOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'MLflowOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'ModelControllerReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'ModelRegistryReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'ModelsAsServiceReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'RayReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'SparkOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainerReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainingOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'TrustyAIReady'}, {'lastTransitionTime': '2026-06-11T09:10:45Z', 'status': 'True', 'type': 'WorkbenchesReady'}], 'observedGeneration': 1, 'phase': 'Not Ready', 'release': {'name': 'OpenShift AI Self-Managed', 'version': '3.4.1'}}. ! --- BVT pytest attempt 2026-06-11T09:10:52.810299+00:00 (/.local/bin/uv run pytest...) --- Bytecode compiled 9467 files in 404ms 2026-06-11T09:10:54.473123Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-06-11T09:10:54.483713Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.483936Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.484112Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.484282Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.484507Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.484691Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.484852Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.485008Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.485176Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.485342Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.485547Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.485703Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.485877Z [info ] [no key 0.00005s] ()  [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.486045Z [info ] [no key 0.00005s] ()  [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.486699Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.486861Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T09:10:54.487378Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-11T09:10:54.498089Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-06-11T09:10:54.613651Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-11T09:10:54.647221Z [info ] Running model registry tests against default namespace [conftest] 2026-06-11T09:10:54.647502Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] ============================= test session starts ============================== platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.14.5', 'Platform': 'Linux-5.14.0-427.115.1.el9_4.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.0.3', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.13.0', 'asyncio': '1.3.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.3.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.1.0'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.13.0, asyncio-1.3.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.3.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.1.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... 2026-06-11T09:10:55.125504Z [info ] Number of unique LMEval tasks with more than 10000 downloads: 35 [tests.ai_safety.lm_eval.utils] name=tests.ai_safety.lm_eval.utils 2026-06-11T09:10:55.148044Z [info ] Number of unique LMEval tasks with more than 1547.0 downloads: 43 [tests.ai_safety.lm_eval.utils] name=tests.ai_safety.lm_eval.utils collected 273 items / 268 deselected / 5 selected tests/ai_safety/evalhub/test_evalhub_deployment.py::test_evalhub_crd_exists  ----------------------------------------------- test_evalhub_crd_exists ----------------------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-11T09:10:55.229014Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() started [sqlalchemy.orm.mapper.Mapper] 2026-06-11T09:10:55.229173Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop id [sqlalchemy.orm.mapper.Mapper] 2026-06-11T09:10:55.229308Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop test_name [sqlalchemy.orm.mapper.Mapper] 2026-06-11T09:10:55.229495Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop start_time [sqlalchemy.orm.mapper.Mapper] 2026-06-11T09:10:55.229621Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() complete [sqlalchemy.orm.mapper.Mapper] 2026-06-11T09:10:55.229741Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T09:10:55.229860Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-11T09:10:55.229993Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T09:10:55.230109Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-11T09:10:55.230260Z [info ] [generated in 0.00015s] ('/home/odh/opendatahub-tests/tests/ai_safety/evalhub/test_evalhub_deployment.py::test_evalhub_crd_exists', 1781169055) [sqlalchemy.engine.Engine] 2026-06-11T09:10:55.230934Z [info ] [generated in 0.00015s] ('/home/odh/opendatahub-tests/tests/ai_safety/evalhub/test_evalhub_deployment.py::test_evalhub_crd_exists', 1781169055) [sqlalchemy.engine.Engine] 2026-06-11T09:10:55.231069Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T09:10:55.231218Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-11T09:10:55.232857Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-06-11T09:10:55.232984Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-06-11T09:10:55.233102Z [info ] Executing session fixture: admin_client [conftest] 2026-06-11T09:10:55.233218Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-11T09:10:55.242874Z [info ] Executing session fixture: dsc_resource [conftest] 2026-06-11T09:10:55.243018Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-06-11T09:10:55.322858Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-06-11T09:10:55.323041Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-06-11T09:10:55.324470Z [info ] Executing session fixture: os_path_environment [conftest] 2026-06-11T09:10:55.324614Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-06-11T09:10:55.324742Z [info ] Executing session fixture: bin_directory [conftest] 2026-06-11T09:10:55.326484Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-06-11T09:10:55.326692Z [info ] kind: ConsoleCLIDownload api version: console.openshift.io/v1 [ocp_resources.resource] 2026-06-11T09:10:55.357820Z [info ] All oc console cli download links: ['https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar'] [utilities.infra] name=utilities.infra 2026-06-11T09:10:55.357942Z [info ] Downloading archive using: url=https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar [utilities.infra] name=utilities.infra 2026-06-11T09:10:57.213548Z [info ] Extract the downloaded archive. [utilities.infra] name=utilities.infra 2026-06-11T09:10:57.403314Z [info ] Downloaded file: ['oc']  [utilities.infra] name=utilities.infra 2026-06-11T09:10:57.438220Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-06-11T09:10:57.436171Z [info ] OC binary path: /tmp/pytest-aVhr8UCP4KXxhkLL3xadSt/bin0/oc [tests.conftest] name=tests.conftest 2026-06-11T09:10:57.436252Z [info ] Adding /tmp/pytest-aVhr8UCP4KXxhkLL3xadSt/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-06-11T09:10:57.438809Z [info ] Executing session fixture: nodes [conftest] 2026-06-11T09:10:57.524056Z [info ] Executing session fixture: dsci_resource [conftest] 2026-06-11T09:10:57.524218Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-11T09:10:57.538449Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-06-11T09:10:57.538610Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-06-11T09:10:57.538821Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-06-11T09:10:57.537754Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-06-11T09:10:57.539100Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-06-11T09:10:57.627969Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-06-11T09:10:57.728775Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsci_status_ready Kwargs: {'dsci_resource': }) [timeout_sampler] 2026-06-11T09:10:57.728036Z [info ] Wait for DSCI default-dsci to be in Ready status. [utilities.infra] name=utilities.infra 2026-06-11T09:10:57.729086Z [info ] Get DSCInitialization default-dsci status [ocp_resources DSCInitialization] 2026-06-11T09:10:57.743704Z [info ] Elapsed time: 0.00011777877807617188 [0:00:00.000118] [timeout_sampler] 2026-06-11T09:10:57.743870Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': }) [timeout_sampler] 2026-06-11T09:10:57.742728Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:10:57.744151Z [info ] Get DataScienceCluster default-dsc status [ocp_resources DataScienceCluster] 2026-06-11T09:11:02.783296Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:11:07.829189Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:11:12.884293Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:11:17.937053Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:11:22.987459Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:11:28.025527Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:11:33.097991Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:11:38.151913Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:11:43.220601Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:11:48.258146Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:11:53.315320Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:11:58.355907Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:12:03.397405Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:12:08.461959Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:12:13.513127Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:12:18.555455Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:12:23.628875Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:12:28.679207Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:12:33.742637Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:12:38.787836Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:12:43.866733Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:12:48.915838Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:12:53.956482Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-11T09:12:59.026285Z [error ] Cluster sanity check failed: Timed Out: 120 Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': } Last exception: ResourceNotReadyError: DSC default-dsc is not ready. Current status: {'components': {'aipipelines': {'managementState': 'Removed'}, 'dashboard': {'managementState': 'Managed'}, 'feastoperator': {'managementState': 'Removed'}, 'kserve': {'managementState': 'Managed', 'releases': [{'name': 'KServe', 'repoUrl': 'https://github.com/kserve/kserve/', 'version': 'v0.17.0'}, {'name': 'vLLM', 'repoUrl': 'https://github.com/vllm-project/vllm', 'version': 'v0.18.0'}, {'name': 'llm-d-inference-scheduler', 'repoUrl': 'https://github.com/llm-d/llm-d-inference-scheduler', 'version': 'v0.7.1'}, {'name': 'llm-d-workload-variant-autoscaler', 'repoUrl': 'https://github.com/llm-d/llm-d-workload-variant-autoscaler', 'version': 'v0.6.0'}]}, 'kueue': {'managementState': 'Removed'}, 'llamastackoperator': {'managementState': 'Removed'}, 'mlflowoperator': {'managementState': 'Removed'}, 'modelregistry': {'managementState': 'Managed', 'registriesNamespace': 'rhoai-model-registries', 'releases': [{'name': 'Kubeflow Model Registry', 'repoUrl': 'https://github.com/kubeflow/model-registry', 'version': 'latest'}, {'name': 'Open Data Hub Model Registry Operator', 'repoUrl': 'https://github.com/opendatahub-io/model-registry-operator', 'version': 'latest'}]}, 'ray': {'managementState': 'Removed'}, 'sparkoperator': {'managementState': 'Removed'}, 'trainer': {'managementState': 'Removed'}, 'trainingoperator': {'managementState': 'Removed'}, 'trustyai': {'managementState': 'Managed', 'releases': [{'name': 'TrustyAI operator', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI service', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-explainability', 'version': 'v0.28.0'}, {'name': 'TrustyAI LMEval driver', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI LMEval job', 'repoUrl': 'https://github.com/EleutherAI/lm-evaluation-harness', 'version': 'v0.4.8'}, {'name': 'TrustyAI Guardrails orchestrator', 'repoUrl': 'https://github.com/foundation-model-stack/fms-guardrails-orchestrator', 'version': '0.9.4'}, {'name': 'TrustyAI builtin detectors', 'repoUrl': 'https://github.com/trustyai-explainability/guardrails-regex-detector', 'version': 'v0.2.0'}, {'name': 'TrustyAI sidecar gateway', 'repoUrl': 'https://github.com/trustyai-explainability/vllm-orchestrator-gateway', 'version': 'v0.2.1'}]}, 'workbenches': {'managementState': 'Managed', 'releases': [{'name': 'Kubeflow Notebook Controller', 'repoUrl': 'https://github.com/kubeflow/kubeflow', 'version': '1.10.0'}], 'workbenchNamespace': 'rhods-notebooks'}}, 'conditions': [{'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'reason': 'Error', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'observedGeneration': 3, 'reason': 'Error', 'status': 'False', 'type': 'ProvisioningSucceeded'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'ComponentsReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'AIPipelinesReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'DashboardReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'FeastOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceDependencies'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceWideEPDependencies'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'KserveReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'KueueReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'LlamaStackOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'MLflowOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'ModelControllerReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'ModelRegistryReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'ModelsAsServiceReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'RayReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'SparkOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainerReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainingOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'TrustyAIReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'WorkbenchesReady'}], 'observedGeneration': 1, 'phase': 'Not Ready', 'release': {'name': 'OpenShift AI Self-Managed', 'version': '3.4.1'}}. [utilities.infra] name=utilities.infra ================ 268 deselected, 1 warning in 124.84s (0:02:04) ================ ================ 268 deselected, 1 warning in 124.84s (0:02:04) ================ ! _pytest.outcomes.Exit: Cluster sanity check failed: Timed Out: 120 Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': } Last exception: ResourceNotReadyError: DSC default-dsc is not ready. Current status: {'components': {'aipipelines': {'managementState': 'Removed'}, 'dashboard': {'managementState': 'Managed'}, 'feastoperator': {'managementState': 'Removed'}, 'kserve': {'managementState': 'Managed', 'releases': [{'name': 'KServe', 'repoUrl': 'https://github.com/kserve/kserve/', 'version': 'v0.17.0'}, {'name': 'vLLM', 'repoUrl': 'https://github.com/vllm-project/vllm', 'version': 'v0.18.0'}, {'name': 'llm-d-inference-scheduler', 'repoUrl': 'https://github.com/llm-d/llm-d-inference-scheduler', 'version': 'v0.7.1'}, {'name': 'llm-d-workload-variant-autoscaler', 'repoUrl': 'https://github.com/llm-d/llm-d-workload-variant-autoscaler', 'version': 'v0.6.0'}]}, 'kueue': {'managementState': 'Removed'}, 'llamastackoperator': {'managementState': 'Removed'}, 'mlflowoperator': {'managementState': 'Removed'}, 'modelregistry': {'managementState': 'Managed', 'registriesNamespace': 'rhoai-model-registries', 'releases': [{'name': 'Kubeflow Model Registry', 'repoUrl': 'https://github.com/kubeflow/model-registry', 'version': 'latest'}, {'name': 'Open Data Hub Model Registry Operator', 'repoUrl': 'https://github.com/opendatahub-io/model-registry-operator', 'version': 'latest'}]}, 'ray': {'managementState': 'Removed'}, 'sparkoperator': {'managementState': 'Removed'}, 'trainer': {'managementState': 'Removed'}, 'trainingoperator': {'managementState': 'Removed'}, 'trustyai': {'managementState': 'Managed', 'releases': [{'name': 'TrustyAI operator', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI service', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-explainability', 'version': 'v0.28.0'}, {'name': 'TrustyAI LMEval driver', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI LMEval job', 'repoUrl': 'https://github.com/EleutherAI/lm-evaluation-harness', 'version': 'v0.4.8'}, {'name': 'TrustyAI Guardrails orchestrator', 'repoUrl': 'https://github.com/foundation-model-stack/fms-guardrails-orchestrator', 'version': '0.9.4'}, {'name': 'TrustyAI builtin detectors', 'repoUrl': 'https://github.com/trustyai-explainability/guardrails-regex-detector', 'version': 'v0.2.0'}, {'name': 'TrustyAI sidecar gateway', 'repoUrl': 'https://github.com/trustyai-explainability/vllm-orchestrator-gateway', 'version': 'v0.2.1'}]}, 'workbenches': {'managementState': 'Managed', 'releases': [{'name': 'Kubeflow Notebook Controller', 'repoUrl': 'https://github.com/kubeflow/kubeflow', 'version': '1.10.0'}], 'workbenchNamespace': 'rhods-notebooks'}}, 'conditions': [{'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'reason': 'Error', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'observedGeneration': 3, 'reason': 'Error', 'status': 'False', 'type': 'ProvisioningSucceeded'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'ComponentsReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'AIPipelinesReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'DashboardReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'FeastOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceDependencies'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceWideEPDependencies'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'KserveReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'KueueReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'LlamaStackOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'MLflowOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'ModelControllerReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'ModelRegistryReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'ModelsAsServiceReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'RayReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'SparkOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainerReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainingOperatorReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'TrustyAIReady'}, {'lastTransitionTime': '2026-06-11T09:12:53Z', 'status': 'True', 'type': 'WorkbenchesReady'}], 'observedGeneration': 1, 'phase': 'Not Ready', 'release': {'name': 'OpenShift AI Self-Managed', 'version': '3.4.1'}}. ! --- BVT pytest attempt 2026-06-11T09:13:07.466132+00:00 (python3 -m pytest...) --- ImportError while loading conftest '/home/odh/opendatahub-tests/conftest.py'. conftest.py:16: in from ocp_resources.cluster_service_version import ClusterServiceVersion E ModuleNotFoundError: No module named 'ocp_resources'