--- BVT pytest attempt 2026-06-14T10:17:03.965287+00:00 (/.local/bin/uv run pytest...) --- 2026-06-14T10:17:05.482068Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-06-14T10:17:05.491372Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.491547Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.491946Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.492080Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.492749Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.492912Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.493059Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.493219Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.493373Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.493891Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.494027Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.494197Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.494335Z [info ] [no key 0.00005s] ()  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.494470Z [info ] [no key 0.00005s] ()  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.494587Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.494699Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.494815Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-06-14T10:17:05.609833Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-06-14T10:17:05.660597Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-14T10:17:05.687811Z [info ] Running model registry tests against default namespace [conftest] 2026-06-14T10:17:05.687970Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] ============================= test session starts ============================== platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.14.5', 'Platform': 'Linux-5.14.0-427.115.1.el9_4.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.0.3', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.13.0', 'asyncio': '1.3.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.3.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.1.0'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.13.0, asyncio-1.3.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.3.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.1.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... collected 105 items / 94 deselected / 11 selected tests/model_serving/maas_billing/maas_api_key/test_api_key_auth_policy_validation.py::TestAuthPolicyApiKeyValidation::test_auth_policy_callback_url_uses_correct_namespace  --------------------------------- test_auth_policy_callback_url_uses_correct_namespace --------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-14T10:17:05.964790Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() started [sqlalchemy.orm.mapper.Mapper] 2026-06-14T10:17:05.964949Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop id [sqlalchemy.orm.mapper.Mapper] 2026-06-14T10:17:05.965063Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop test_name [sqlalchemy.orm.mapper.Mapper] 2026-06-14T10:17:05.965591Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop start_time [sqlalchemy.orm.mapper.Mapper] 2026-06-14T10:17:05.965717Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() complete [sqlalchemy.orm.mapper.Mapper] 2026-06-14T10:17:05.965837Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.965955Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.966071Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.966573Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.966737Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/maas_api_key/test_api_key_auth_policy_validation.py::test_auth_policy_callback_url_uses_correct_namespace', 1781432225) [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.966886Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/maas_api_key/test_api_key_auth_policy_validation.py::test_auth_policy_callback_url_uses_correct_namespace', 1781432225) [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.967010Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.967217Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-14T10:17:05.968258Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-06-14T10:17:05.968375Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-06-14T10:17:05.968493Z [info ] Executing session fixture: admin_client [conftest] 2026-06-14T10:17:05.968614Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-14T10:17:05.972036Z [info ] Executing session fixture: dsc_resource [conftest] 2026-06-14T10:17:05.972197Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-06-14T10:17:06.035815Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-06-14T10:17:06.035953Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-06-14T10:17:06.037029Z [info ] Executing session fixture: os_path_environment [conftest] 2026-06-14T10:17:06.037289Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-06-14T10:17:06.037448Z [info ] Executing session fixture: bin_directory [conftest] 2026-06-14T10:17:06.038602Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-06-14T10:17:06.051247Z [info ] kind: ConsoleCLIDownload api version: console.openshift.io/v1 [ocp_resources.resource] 2026-06-14T10:17:06.075951Z [info ] All oc console cli download links: ['https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar'] [utilities.infra] name=utilities.infra 2026-06-14T10:17:06.076048Z [info ] Downloading archive using: url=https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar [utilities.infra] name=utilities.infra 2026-06-14T10:17:26.581121Z [info ] Extract the downloaded archive. [utilities.infra] name=utilities.infra 2026-06-14T10:17:26.698867Z [info ] Downloaded file: ['oc']  [utilities.infra] name=utilities.infra 2026-06-14T10:17:26.724763Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-06-14T10:17:26.722406Z [info ] OC binary path: /tmp/pytest-D6ThnD5RZDKXTAMRMnYAX2/bin0/oc [tests.conftest] name=tests.conftest 2026-06-14T10:17:26.722500Z [info ] Adding /tmp/pytest-D6ThnD5RZDKXTAMRMnYAX2/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-06-14T10:17:26.725414Z [info ] Executing session fixture: nodes [conftest] 2026-06-14T10:17:27.621057Z [info ] Executing session fixture: dsci_resource [conftest] 2026-06-14T10:17:27.621292Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-14T10:17:27.634889Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-06-14T10:17:27.635052Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-06-14T10:17:27.635267Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-06-14T10:17:27.634206Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-06-14T10:17:27.635573Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-06-14T10:17:27.720563Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-06-14T10:17:27.806477Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsci_status_ready Kwargs: {'dsci_resource': }) [timeout_sampler] 2026-06-14T10:17:27.805892Z [info ] Wait for DSCI default-dsci to be in Ready status. [utilities.infra] name=utilities.infra 2026-06-14T10:17:27.806758Z [info ] Get DSCInitialization default-dsci status [ocp_resources DSCInitialization] 2026-06-14T10:17:27.820240Z [info ] Elapsed time: 0.0001366138458251953 [0:00:00.000137] [timeout_sampler] 2026-06-14T10:17:27.820405Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': }) [timeout_sampler] 2026-06-14T10:17:27.819719Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:17:27.820668Z [info ] Get DataScienceCluster default-dsc status [ocp_resources DataScienceCluster] 2026-06-14T10:17:32.853528Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:17:37.904999Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:17:42.977033Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:17:48.012614Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:17:53.054244Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:17:58.100620Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:18:03.136545Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:18:08.185405Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:18:13.222887Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:18:18.270845Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:18:23.314206Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:18:28.354978Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:18:33.391362Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:18:38.440537Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:18:43.483516Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:18:48.528798Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:18:53.562240Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:18:58.599039Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:19:03.635307Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:19:08.679353Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:19:13.717859Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:19:18.752589Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:19:23.788072Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:19:28.824622Z [error ] Cluster sanity check failed: Timed Out: 120 Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': } Last exception: ResourceNotReadyError: DSC default-dsc is not ready. Current status: {'components': {'aipipelines': {'managementState': 'Removed'}, 'dashboard': {'managementState': 'Managed'}, 'feastoperator': {'managementState': 'Removed'}, 'kserve': {'managementState': 'Managed', 'releases': [{'name': 'KServe', 'repoUrl': 'https://github.com/kserve/kserve/', 'version': 'v0.17.0'}, {'name': 'vLLM', 'repoUrl': 'https://github.com/vllm-project/vllm', 'version': 'v0.18.0'}, {'name': 'llm-d-inference-scheduler', 'repoUrl': 'https://github.com/llm-d/llm-d-inference-scheduler', 'version': 'v0.7.1'}, {'name': 'llm-d-workload-variant-autoscaler', 'repoUrl': 'https://github.com/llm-d/llm-d-workload-variant-autoscaler', 'version': 'v0.6.0'}]}, 'kueue': {'managementState': 'Removed'}, 'llamastackoperator': {'managementState': 'Removed'}, 'mlflowoperator': {'managementState': 'Removed'}, 'modelregistry': {'managementState': 'Removed'}, 'ray': {'managementState': 'Managed', 'releases': [{'name': 'KubeRay', 'repoUrl': 'https://github.com/ray-project/kuberay', 'version': '1.4.2'}]}, 'sparkoperator': {'managementState': 'Removed'}, 'trainer': {'managementState': 'Removed'}, 'trainingoperator': {'managementState': 'Removed'}, 'trustyai': {'managementState': 'Managed', 'releases': [{'name': 'TrustyAI operator', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI service', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-explainability', 'version': 'v0.28.0'}, {'name': 'TrustyAI LMEval driver', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI LMEval job', 'repoUrl': 'https://github.com/EleutherAI/lm-evaluation-harness', 'version': 'v0.4.8'}, {'name': 'TrustyAI Guardrails orchestrator', 'repoUrl': 'https://github.com/foundation-model-stack/fms-guardrails-orchestrator', 'version': '0.9.4'}, {'name': 'TrustyAI builtin detectors', 'repoUrl': 'https://github.com/trustyai-explainability/guardrails-regex-detector', 'version': 'v0.2.0'}, {'name': 'TrustyAI sidecar gateway', 'repoUrl': 'https://github.com/trustyai-explainability/vllm-orchestrator-gateway', 'version': 'v0.2.1'}]}, 'workbenches': {'managementState': 'Managed', 'releases': [{'name': 'Kubeflow Notebook Controller', 'repoUrl': 'https://github.com/kubeflow/kubeflow', 'version': '1.10.0'}], 'workbenchNamespace': 'rhods-notebooks'}}, 'conditions': [{'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'reason': 'Error', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'observedGeneration': 3, 'reason': 'Error', 'status': 'False', 'type': 'ProvisioningSucceeded'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'ComponentsReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'AIPipelinesReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'DashboardReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'FeastOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceDependencies'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceWideEPDependencies'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'KserveReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'KueueReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'LlamaStackOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'MLflowOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'ModelControllerReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'ModelRegistryReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'ModelsAsServiceReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'RayReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'SparkOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainerReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainingOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'TrustyAIReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'WorkbenchesReady'}], 'observedGeneration': 1, 'phase': 'Not Ready', 'release': {'name': 'OpenShift AI Self-Managed', 'version': '3.4.0'}}. [utilities.infra] name=utilities.infra ================ 94 deselected, 1 warning in 143.57s (0:02:23) ================= ================ 94 deselected, 1 warning in 143.57s (0:02:23) ================= ! _pytest.outcomes.Exit: Cluster sanity check failed: Timed Out: 120 Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': } Last exception: ResourceNotReadyError: DSC default-dsc is not ready. Current status: {'components': {'aipipelines': {'managementState': 'Removed'}, 'dashboard': {'managementState': 'Managed'}, 'feastoperator': {'managementState': 'Removed'}, 'kserve': {'managementState': 'Managed', 'releases': [{'name': 'KServe', 'repoUrl': 'https://github.com/kserve/kserve/', 'version': 'v0.17.0'}, {'name': 'vLLM', 'repoUrl': 'https://github.com/vllm-project/vllm', 'version': 'v0.18.0'}, {'name': 'llm-d-inference-scheduler', 'repoUrl': 'https://github.com/llm-d/llm-d-inference-scheduler', 'version': 'v0.7.1'}, {'name': 'llm-d-workload-variant-autoscaler', 'repoUrl': 'https://github.com/llm-d/llm-d-workload-variant-autoscaler', 'version': 'v0.6.0'}]}, 'kueue': {'managementState': 'Removed'}, 'llamastackoperator': {'managementState': 'Removed'}, 'mlflowoperator': {'managementState': 'Removed'}, 'modelregistry': {'managementState': 'Removed'}, 'ray': {'managementState': 'Managed', 'releases': [{'name': 'KubeRay', 'repoUrl': 'https://github.com/ray-project/kuberay', 'version': '1.4.2'}]}, 'sparkoperator': {'managementState': 'Removed'}, 'trainer': {'managementState': 'Removed'}, 'trainingoperator': {'managementState': 'Removed'}, 'trustyai': {'managementState': 'Managed', 'releases': [{'name': 'TrustyAI operator', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI service', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-explainability', 'version': 'v0.28.0'}, {'name': 'TrustyAI LMEval driver', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI LMEval job', 'repoUrl': 'https://github.com/EleutherAI/lm-evaluation-harness', 'version': 'v0.4.8'}, {'name': 'TrustyAI Guardrails orchestrator', 'repoUrl': 'https://github.com/foundation-model-stack/fms-guardrails-orchestrator', 'version': '0.9.4'}, {'name': 'TrustyAI builtin detectors', 'repoUrl': 'https://github.com/trustyai-explainability/guardrails-regex-detector', 'version': 'v0.2.0'}, {'name': 'TrustyAI sidecar gateway', 'repoUrl': 'https://github.com/trustyai-explainability/vllm-orchestrator-gateway', 'version': 'v0.2.1'}]}, 'workbenches': {'managementState': 'Managed', 'releases': [{'name': 'Kubeflow Notebook Controller', 'repoUrl': 'https://github.com/kubeflow/kubeflow', 'version': '1.10.0'}], 'workbenchNamespace': 'rhods-notebooks'}}, 'conditions': [{'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'reason': 'Error', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'observedGeneration': 3, 'reason': 'Error', 'status': 'False', 'type': 'ProvisioningSucceeded'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'ComponentsReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'AIPipelinesReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'DashboardReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'FeastOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceDependencies'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceWideEPDependencies'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'KserveReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'KueueReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'LlamaStackOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'MLflowOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'ModelControllerReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'ModelRegistryReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'ModelsAsServiceReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'RayReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'SparkOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainerReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainingOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'TrustyAIReady'}, {'lastTransitionTime': '2026-06-14T10:19:23Z', 'status': 'True', 'type': 'WorkbenchesReady'}], 'observedGeneration': 1, 'phase': 'Not Ready', 'release': {'name': 'OpenShift AI Self-Managed', 'version': '3.4.0'}}. ! --- BVT pytest attempt 2026-06-14T10:19:29.806453+00:00 (/.local/bin/uv run pytest...) --- Bytecode compiled 9467 files in 406ms 2026-06-14T10:19:31.397427Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-06-14T10:19:31.406530Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.406967Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.407849Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.408012Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.408186Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.408344Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.408639Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.408785Z [info ] PRAGMA temp.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.408964Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.409161Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.409497Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.411775Z [info ]  CREATE TABLE "OpenDataHubTestTable" ( id INTEGER NOT NULL, test_name VARCHAR(500) NOT NULL, start_time INTEGER NOT NULL, PRIMARY KEY (id) )  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.411977Z [info ] [no key 0.00005s] ()  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.412157Z [info ] [no key 0.00005s] ()  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.412292Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.412457Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.412587Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-14T10:19:31.418978Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] /home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.ods-qe-psi-23.osp.rh-ods.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2026-06-14T10:19:31.502857Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-14T10:19:31.532132Z [info ] Running model registry tests against default namespace [conftest] 2026-06-14T10:19:31.532357Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] ============================= test session starts ============================== platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.14.5', 'Platform': 'Linux-5.14.0-427.115.1.el9_4.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.0.3', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.13.0', 'asyncio': '1.3.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.3.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.1.0'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.13.0, asyncio-1.3.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.3.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.1.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... collected 105 items / 94 deselected / 11 selected tests/model_serving/maas_billing/maas_api_key/test_api_key_auth_policy_validation.py::TestAuthPolicyApiKeyValidation::test_auth_policy_callback_url_uses_correct_namespace  --------------------------------- test_auth_policy_callback_url_uses_correct_namespace --------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-14T10:19:31.697053Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() started [sqlalchemy.orm.mapper.Mapper] 2026-06-14T10:19:31.697251Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop id [sqlalchemy.orm.mapper.Mapper] 2026-06-14T10:19:31.697386Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop test_name [sqlalchemy.orm.mapper.Mapper] 2026-06-14T10:19:31.697521Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop start_time [sqlalchemy.orm.mapper.Mapper] 2026-06-14T10:19:31.698163Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() complete [sqlalchemy.orm.mapper.Mapper] 2026-06-14T10:19:31.698307Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.698446Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.698572Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.698692Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.698844Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/maas_api_key/test_api_key_auth_policy_validation.py::test_auth_policy_callback_url_uses_correct_namespace', 1781432371) [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.698985Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/maas_api_key/test_api_key_auth_policy_validation.py::test_auth_policy_callback_url_uses_correct_namespace', 1781432371) [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.699616Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.699744Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-14T10:19:31.701159Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-06-14T10:19:31.701346Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-06-14T10:19:31.703590Z [info ] Executing session fixture: admin_client [conftest] 2026-06-14T10:19:31.703734Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-14T10:19:31.709490Z [info ] Executing session fixture: dsc_resource [conftest] 2026-06-14T10:19:31.709617Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-06-14T10:19:31.779448Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-06-14T10:19:31.779942Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-06-14T10:19:31.780724Z [info ] Executing session fixture: os_path_environment [conftest] 2026-06-14T10:19:31.780861Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-06-14T10:19:31.780989Z [info ] Executing session fixture: bin_directory [conftest] 2026-06-14T10:19:31.782018Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-06-14T10:19:31.782178Z [info ] kind: ConsoleCLIDownload api version: console.openshift.io/v1 [ocp_resources.resource] 2026-06-14T10:19:31.807250Z [info ] All oc console cli download links: ['https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar'] [utilities.infra] name=utilities.infra 2026-06-14T10:19:31.807339Z [info ] Downloading archive using: url=https://downloads-openshift-console.apps.ods-qe-psi-23.osp.rh-ods.com/amd64/linux/oc.tar [utilities.infra] name=utilities.infra 2026-06-14T10:19:38.849304Z [info ] Extract the downloaded archive. [utilities.infra] name=utilities.infra 2026-06-14T10:19:39.001465Z [info ] Downloaded file: ['oc']  [utilities.infra] name=utilities.infra 2026-06-14T10:19:39.030597Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-06-14T10:19:39.027215Z [info ] OC binary path: /tmp/pytest-Cm7oSTCwKeygy3M8rKRWmy/bin0/oc [tests.conftest] name=tests.conftest 2026-06-14T10:19:39.027292Z [info ] Adding /tmp/pytest-Cm7oSTCwKeygy3M8rKRWmy/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-06-14T10:19:39.031046Z [info ] Executing session fixture: nodes [conftest] 2026-06-14T10:19:39.084399Z [info ] Executing session fixture: dsci_resource [conftest] 2026-06-14T10:19:39.084554Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-14T10:19:39.098004Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-06-14T10:19:39.098245Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-06-14T10:19:39.098397Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-06-14T10:19:39.097361Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-06-14T10:19:39.098625Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-06-14T10:19:39.180001Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-06-14T10:19:39.267461Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsci_status_ready Kwargs: {'dsci_resource': }) [timeout_sampler] 2026-06-14T10:19:39.266895Z [info ] Wait for DSCI default-dsci to be in Ready status. [utilities.infra] name=utilities.infra 2026-06-14T10:19:39.267748Z [info ] Get DSCInitialization default-dsci status [ocp_resources DSCInitialization] 2026-06-14T10:19:39.280622Z [info ] Elapsed time: 8.869171142578125e-05 [0:00:00.000089] [timeout_sampler] 2026-06-14T10:19:39.280768Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': }) [timeout_sampler] 2026-06-14T10:19:39.280068Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:19:39.281026Z [info ] Get DataScienceCluster default-dsc status [ocp_resources DataScienceCluster] 2026-06-14T10:19:44.313068Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:19:49.351751Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:19:54.385878Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:19:59.423064Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:20:04.459390Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:20:09.507314Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:20:14.550564Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:20:19.593606Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:20:24.627073Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:20:29.666625Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:20:34.702564Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:20:39.747033Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:20:44.789219Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:20:49.825662Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:20:54.858428Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:20:59.891727Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:21:04.936501Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:21:09.974127Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:21:15.016537Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:21:20.064362Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:21:25.101752Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:21:30.140341Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:21:35.182590Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-14T10:21:40.215722Z [error ] Cluster sanity check failed: Timed Out: 120 Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': } Last exception: ResourceNotReadyError: DSC default-dsc is not ready. Current status: {'components': {'aipipelines': {'managementState': 'Removed'}, 'dashboard': {'managementState': 'Managed'}, 'feastoperator': {'managementState': 'Removed'}, 'kserve': {'managementState': 'Managed', 'releases': [{'name': 'KServe', 'repoUrl': 'https://github.com/kserve/kserve/', 'version': 'v0.17.0'}, {'name': 'vLLM', 'repoUrl': 'https://github.com/vllm-project/vllm', 'version': 'v0.18.0'}, {'name': 'llm-d-inference-scheduler', 'repoUrl': 'https://github.com/llm-d/llm-d-inference-scheduler', 'version': 'v0.7.1'}, {'name': 'llm-d-workload-variant-autoscaler', 'repoUrl': 'https://github.com/llm-d/llm-d-workload-variant-autoscaler', 'version': 'v0.6.0'}]}, 'kueue': {'managementState': 'Removed'}, 'llamastackoperator': {'managementState': 'Removed'}, 'mlflowoperator': {'managementState': 'Removed'}, 'modelregistry': {'managementState': 'Removed'}, 'ray': {'managementState': 'Managed', 'releases': [{'name': 'KubeRay', 'repoUrl': 'https://github.com/ray-project/kuberay', 'version': '1.4.2'}]}, 'sparkoperator': {'managementState': 'Removed'}, 'trainer': {'managementState': 'Removed'}, 'trainingoperator': {'managementState': 'Removed'}, 'trustyai': {'managementState': 'Managed', 'releases': [{'name': 'TrustyAI operator', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI service', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-explainability', 'version': 'v0.28.0'}, {'name': 'TrustyAI LMEval driver', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI LMEval job', 'repoUrl': 'https://github.com/EleutherAI/lm-evaluation-harness', 'version': 'v0.4.8'}, {'name': 'TrustyAI Guardrails orchestrator', 'repoUrl': 'https://github.com/foundation-model-stack/fms-guardrails-orchestrator', 'version': '0.9.4'}, {'name': 'TrustyAI builtin detectors', 'repoUrl': 'https://github.com/trustyai-explainability/guardrails-regex-detector', 'version': 'v0.2.0'}, {'name': 'TrustyAI sidecar gateway', 'repoUrl': 'https://github.com/trustyai-explainability/vllm-orchestrator-gateway', 'version': 'v0.2.1'}]}, 'workbenches': {'managementState': 'Managed', 'releases': [{'name': 'Kubeflow Notebook Controller', 'repoUrl': 'https://github.com/kubeflow/kubeflow', 'version': '1.10.0'}], 'workbenchNamespace': 'rhods-notebooks'}}, 'conditions': [{'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'reason': 'Error', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'observedGeneration': 3, 'reason': 'Error', 'status': 'False', 'type': 'ProvisioningSucceeded'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'ComponentsReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'AIPipelinesReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'DashboardReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'FeastOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceDependencies'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceWideEPDependencies'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'KserveReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'KueueReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'LlamaStackOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'MLflowOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'ModelControllerReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'ModelRegistryReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'ModelsAsServiceReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'RayReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'SparkOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainerReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainingOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'TrustyAIReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'WorkbenchesReady'}], 'observedGeneration': 1, 'phase': 'Not Ready', 'release': {'name': 'OpenShift AI Self-Managed', 'version': '3.4.0'}}. [utilities.infra] name=utilities.infra ================ 94 deselected, 1 warning in 129.11s (0:02:09) ================= ================ 94 deselected, 1 warning in 129.11s (0:02:09) ================= ! _pytest.outcomes.Exit: Cluster sanity check failed: Timed Out: 120 Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': } Last exception: ResourceNotReadyError: DSC default-dsc is not ready. Current status: {'components': {'aipipelines': {'managementState': 'Removed'}, 'dashboard': {'managementState': 'Managed'}, 'feastoperator': {'managementState': 'Removed'}, 'kserve': {'managementState': 'Managed', 'releases': [{'name': 'KServe', 'repoUrl': 'https://github.com/kserve/kserve/', 'version': 'v0.17.0'}, {'name': 'vLLM', 'repoUrl': 'https://github.com/vllm-project/vllm', 'version': 'v0.18.0'}, {'name': 'llm-d-inference-scheduler', 'repoUrl': 'https://github.com/llm-d/llm-d-inference-scheduler', 'version': 'v0.7.1'}, {'name': 'llm-d-workload-variant-autoscaler', 'repoUrl': 'https://github.com/llm-d/llm-d-workload-variant-autoscaler', 'version': 'v0.6.0'}]}, 'kueue': {'managementState': 'Removed'}, 'llamastackoperator': {'managementState': 'Removed'}, 'mlflowoperator': {'managementState': 'Removed'}, 'modelregistry': {'managementState': 'Removed'}, 'ray': {'managementState': 'Managed', 'releases': [{'name': 'KubeRay', 'repoUrl': 'https://github.com/ray-project/kuberay', 'version': '1.4.2'}]}, 'sparkoperator': {'managementState': 'Removed'}, 'trainer': {'managementState': 'Removed'}, 'trainingoperator': {'managementState': 'Removed'}, 'trustyai': {'managementState': 'Managed', 'releases': [{'name': 'TrustyAI operator', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI service', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-explainability', 'version': 'v0.28.0'}, {'name': 'TrustyAI LMEval driver', 'repoUrl': 'https://github.com/trustyai-explainability/trustyai-service-operator', 'version': 'v1.37.0'}, {'name': 'TrustyAI LMEval job', 'repoUrl': 'https://github.com/EleutherAI/lm-evaluation-harness', 'version': 'v0.4.8'}, {'name': 'TrustyAI Guardrails orchestrator', 'repoUrl': 'https://github.com/foundation-model-stack/fms-guardrails-orchestrator', 'version': '0.9.4'}, {'name': 'TrustyAI builtin detectors', 'repoUrl': 'https://github.com/trustyai-explainability/guardrails-regex-detector', 'version': 'v0.2.0'}, {'name': 'TrustyAI sidecar gateway', 'repoUrl': 'https://github.com/trustyai-explainability/vllm-orchestrator-gateway', 'version': 'v0.2.1'}]}, 'workbenches': {'managementState': 'Managed', 'releases': [{'name': 'Kubeflow Notebook Controller', 'repoUrl': 'https://github.com/kubeflow/kubeflow', 'version': '1.10.0'}], 'workbenchNamespace': 'rhods-notebooks'}}, 'conditions': [{'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'reason': 'Error', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'failure deploying resource /default-kserve: apply failed ' 'components.platform.opendatahub.io/v1alpha1, Kind=Kserve: unable ' 'to patch components.platform.opendatahub.io/v1alpha1, Kind=Kserve ' 'default-kserve: Kserve.components.platform.opendatahub.io ' '"default-kserve" is invalid: [spec.modelsAsService: Invalid ' 'value: "null": spec.modelsAsService in body must be of type ' 'object: "null", : Invalid value: "null": some validation ' 'rules were not checked because the object was invalid; correct ' 'the existing errors to complete validation]', 'observedGeneration': 3, 'reason': 'Error', 'status': 'False', 'type': 'ProvisioningSucceeded'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'ComponentsReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'AIPipelinesReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'DashboardReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'FeastOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceDependencies'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'KserveLLMInferenceServiceWideEPDependencies'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'KserveReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'KueueReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'LlamaStackOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'MLflowOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'ModelControllerReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'ModelRegistryReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'ModelsAsServiceReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'RayReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'SparkOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainerReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'message': 'Component ManagementState is set to Removed', 'reason': 'Removed', 'severity': 'Info', 'status': 'False', 'type': 'TrainingOperatorReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'TrustyAIReady'}, {'lastTransitionTime': '2026-06-14T10:21:34Z', 'status': 'True', 'type': 'WorkbenchesReady'}], 'observedGeneration': 1, 'phase': 'Not Ready', 'release': {'name': 'OpenShift AI Self-Managed', 'version': '3.4.0'}}. ! --- BVT pytest attempt 2026-06-14T10:21:48.128426+00:00 (python3 -m pytest...) --- ImportError while loading conftest '/home/odh/opendatahub-tests/conftest.py'. conftest.py:16: in from ocp_resources.cluster_service_version import ClusterServiceVersion E ModuleNotFoundError: No module named 'ocp_resources'