
--- pytest attempt 2026-11-01T13:53:39.777728+00:00 (/.local/bin/uv --directory /home/odh/opendatahub-tests...) ---
[2m2026-11-01T13:53:42.069749Z[0m [[32m[1minfo     [0m] [1mWriting tests log to /home/odh/opendatahub-tests/results/pytest-tests.log[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:42.070154Z[0m [[32m[1minfo     [0m] [1mTrying to get client via new_client_from_config[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:42.170213Z[0m [[32m[1minfo     [0m] [1mkind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:42.224736Z[0m [[32m[1minfo     [0m] [1mkind: DSCInitialization api version: dscinitialization.opendatahub.io/v2[0m [[0m[1m[34mocp_resources.resource[0m][0m
============================= test session starts ==============================
platform linux -- Python 3.14.8, pytest-9.1.1, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.14.8', 'Platform': 'Linux-5.14.0-570.142.1.el9_6.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.1.1', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.14.2', 'langsmith': '0.10.10', 'asyncio': '1.4.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.5.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.5.3'}}
rootdir: /home/odh/opendatahub-tests
configfile: pytest.ini
plugins: anyio-4.14.2, langsmith-0.10.10, asyncio-1.4.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.5.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.5.3
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collecting ... collected 218 items / 208 deselected / 10 selected

tests/model_serving/maas_billing/body_base_routing/pre_auth_model_header/test_bbr_envoy_filter.py::TestBBREnvoyFilter::test_bbr_envoy_filter_has_pre_and_post_auth_stages [1m
---------------------------------- test_bbr_envoy_filter_has_pre_and_post_auth_stages ----------------------------------[0m
[1m-------------------------------------------------------- SETUP --------------------------------------------------------[0m
[2m2026-11-01T13:53:42.793394Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: event_loop_policy[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:42.797101Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: _syrupy_apply_ide_patches[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:42.797293Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: admin_client[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:42.797437Z[0m [[32m[1minfo     [0m] [1mTrying to get client via new_client_from_config[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:42.797570Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: dsc_resource[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:42.797704Z[0m [[32m[1minfo     [0m] [1mkind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:42.860796Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: tmp_path_factory[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:42.860980Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: tests_tmp_dir[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:42.861926Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: os_path_environment[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:42.862059Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: tmpdir_factory[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:42.862225Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: bin_directory[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:42.863335Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: oc_binary_path[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:42.862590Z[0m [[33m[1mwarning  [0m] [1mUsing previously installed: /workspace/tests-shared/tests-payload/.tools/bin/oc[0m [[0m[1m[34mtests.conftest[0m][0m [36mname[0m=[35mtests.conftest[0m
[2m2026-11-01T13:53:42.863706Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: bin_directory_to_os_path[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:42.862724Z[0m [[32m[1minfo     [0m] [1mOC binary path: /workspace/tests-shared/tests-payload/.tools/bin/oc[0m [[0m[1m[34mtests.conftest[0m][0m [36mname[0m=[35mtests.conftest[0m
[2m2026-11-01T13:53:42.862766Z[0m [[32m[1minfo     [0m] [1mAdding /tmp/pytest-EZdqAP7yk242k3JNnYwSaA/bin0 to $PATH[0m [[0m[1m[34mtests.conftest[0m][0m [36mname[0m=[35mtests.conftest[0m
[2m2026-11-01T13:53:42.864081Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: nodes[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:43.858656Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: dsci_resource[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:43.858829Z[0m [[32m[1minfo     [0m] [1mkind: DSCInitialization api version: dscinitialization.opendatahub.io/v2[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:43.873503Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: record_testsuite_property[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:43.873698Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: junitxml_plugin[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:43.873865Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: cluster_sanity_scope_session[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:43.872698Z[0m [[32m[1minfo     [0m] [1mCheck cluster sanity.         [0m [[0m[1m[34mutilities.infra[0m][0m [36mname[0m=[35mutilities.infra[0m
[2m2026-11-01T13:53:43.874151Z[0m [[32m[1minfo     [0m] [1mVerify all nodes are in a healthy condition.[0m [[0m[1m[34mocp_utilities.infra[0m][0m
[2m2026-11-01T13:53:43.917630Z[0m [[32m[1minfo     [0m] [1mVerify all nodes are schedulable.[0m [[0m[1m[34mocp_utilities.infra[0m][0m
[2m2026-11-01T13:53:43.963715Z[0m [[32m[1minfo     [0m] [1mWaiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsci_status_ready  Kwargs: {'dsci_resource': <ocp_resources.dsc_initialization.DSCInitialization object at 0x7fea4c9e6d50>})[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:43.963154Z[0m [[32m[1minfo     [0m] [1mWait for DSCI default-dsci to be in Ready status.[0m [[0m[1m[34mutilities.infra[0m][0m [36mname[0m=[35mutilities.infra[0m
[2m2026-11-01T13:53:43.963997Z[0m [[32m[1minfo     [0m] [1mGet DSCInitialization default-dsci status[0m [[0m[1m[34mocp_resources DSCInitialization[0m][0m
[2m2026-11-01T13:53:43.978432Z[0m [success  [0m] [1mFunction: utilities.infra.wait_for_dsci_status_ready succeeded after 0.0001437664031982422 [0:00:00.000144].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:43.978842Z[0m [[32m[1minfo     [0m] [1mWaiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsc_status_ready  Kwargs: {'dsc_resource': <ocp_resources.data_science_cluster.DataScienceCluster object at 0x7fea4c9e65d0>})[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:43.977789Z[0m [[32m[1minfo     [0m] [1mWait for DSC default-dsc are Ready.[0m [[0m[1m[34mutilities.infra[0m][0m [36mname[0m=[35mutilities.infra[0m
[2m2026-11-01T13:53:43.979202Z[0m [[32m[1minfo     [0m] [1mGet DataScienceCluster default-dsc status[0m [[0m[1m[34mocp_resources DataScienceCluster[0m][0m
[2m2026-11-01T13:53:43.995525Z[0m [success  [0m] [1mFunction: utilities.infra.wait_for_dsc_status_ready succeeded after 5.9604644775390625e-05 [0:00:00.000060].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:43.995677Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: autouse_fixtures[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:43.995823Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: maas_gateway_api_hostname[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:43.995965Z[0m [[32m[1minfo     [0m] [1mkind: Ingress api version: config.openshift.io/v1[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:44.025461Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: maas_gateway_api[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.049718Z[0m [[32m[1minfo     [0m] [1mkind: Gateway api version: gateway.networking.k8s.io/v1[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:44.064875Z[0m [[32m[1minfo     [0m] [1mReusing existing gateway openshift-ingress/maas-default-gateway[0m [[0m[1m[34mconftest[0m][0m [36mname[0m=[35mconftest[0m
[2m2026-11-01T13:53:44.066072Z[0m [[32m[1minfo     [0m] [1mWait for Gateway/maas-default-gateway's 'Programmed' condition to be 'True'[0m [[0m[1m[34mocp_resources Gateway[0m][0m
[2m2026-11-01T13:53:44.066282Z[0m [[32m[1minfo     [0m] [1mWait until Gateway maas-default-gateway is created[0m [[0m[1m[34mocp_resources Gateway[0m][0m
[2m2026-11-01T13:53:44.066422Z[0m [[32m[1minfo     [0m] [1mWaiting for 300 seconds [0:05:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.081364Z[0m [success  [0m] [1mFunction: lambda: self.exists succeeded after 6.29425048828125e-05 [0:00:00.000063].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.081535Z[0m [[32m[1minfo     [0m] [1mWaiting for 299.984361410141 seconds [0:04:59.984361], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.096819Z[0m [success  [0m] [1mFunction: lambda: self.instance succeeded after 4.744529724121094e-05 [0:00:00.000047].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.096973Z[0m [[32m[1minfo     [0m] [1mResourceEdit: Backing up old data[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:44.112987Z[0m [[32m[1minfo     [0m] [1mResourceEdits: Updating data for resource Gateway maas-default-gateway[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:44.113137Z[0m [[32m[1minfo     [0m] [1mUpdate Gateway maas-default-gateway:
{'metadata': {'annotations': {'security.opendatahub.io/authorino-tls-bootstrap': 'true'}, 'name': 'maas-default-gateway'}}[0m [[0m[1m[34mocp_resources Gateway[0m][0m
[2m2026-11-01T13:53:44.137045Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: pytestconfig[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.137314Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: use_unprivileged_client[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.137447Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: kubconfig_filepath[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.137566Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: cluster_authentication[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.137682Z[0m [[32m[1minfo     [0m] [1mkind: Authentication api version: config.openshift.io/v1[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:44.151704Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: is_byoidc[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.167418Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: non_admin_user_password[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.167656Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: unprivileged_client[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.166112Z[0m [[33m[1mwarning  [0m] [1mUnprivileged client is not enabled, using admin client[0m [[0m[1m[34mtests.conftest[0m][0m [36mname[0m=[35mtests.conftest[0m
[2m2026-11-01T13:53:44.167906Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: maas_subscription_namespace[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.180564Z[0m [[32m[1minfo     [0m] [1mNamespace models-as-a-service already exists, reusing it[0m [[0m[1m[34mconftest[0m][0m [36mname[0m=[35mconftest[0m
[2m2026-11-01T13:53:44.181794Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: maas_subscription_controller_enabled_latest[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.181919Z[0m [[32m[1minfo     [0m] [1mkind: CustomResourceDefinition api version: apiextensions.k8s.io/v1[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:44.281094Z[0m [[32m[1minfo     [0m] [1mResourceEdit: Backing up old data[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:44.297990Z[0m [[32m[1minfo     [0m] [1mResourceEdits: Updating data for resource DataScienceCluster default-dsc[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:44.298162Z[0m [[32m[1minfo     [0m] [1mUpdate DataScienceCluster default-dsc:
{'spec': {'components': {'kserve': {'modelsAsService': {'managementState': 'Managed'}}}}, 'metadata': {'name': 'default-dsc'}}[0m [[0m[1m[34mocp_resources DataScienceCluster[0m][0m
[2m2026-11-01T13:53:44.429425Z[0m [[32m[1minfo     [0m] [1mWait for DataScienceCluster/default-dsc's 'ModelsAsServiceReady' condition to be 'True'[0m [[0m[1m[34mocp_resources DataScienceCluster[0m][0m
[2m2026-11-01T13:53:44.429672Z[0m [[32m[1minfo     [0m] [1mWait until DataScienceCluster default-dsc is created[0m [[0m[1m[34mocp_resources DataScienceCluster[0m][0m
[2m2026-11-01T13:53:44.429808Z[0m [[32m[1minfo     [0m] [1mWaiting for 900 seconds [0:15:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.446609Z[0m [success  [0m] [1mFunction: lambda: self.exists succeeded after 6.127357482910156e-05 [0:00:00.000061].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.446789Z[0m [[32m[1minfo     [0m] [1mWaiting for 899.9826140403748 seconds [0:14:59.982614], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.462433Z[0m [success  [0m] [1mFunction: lambda: self.instance succeeded after 3.8623809814453125e-05 [0:00:00.000039].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.462599Z[0m [[32m[1minfo     [0m] [1mWait for DataScienceCluster/default-dsc's 'Ready' condition to be 'True'[0m [[0m[1m[34mocp_resources DataScienceCluster[0m][0m
[2m2026-11-01T13:53:44.462765Z[0m [[32m[1minfo     [0m] [1mWait until DataScienceCluster default-dsc is created[0m [[0m[1m[34mocp_resources DataScienceCluster[0m][0m
[2m2026-11-01T13:53:44.462935Z[0m [[32m[1minfo     [0m] [1mWaiting for 600 seconds [0:10:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.478979Z[0m [success  [0m] [1mFunction: lambda: self.exists succeeded after 3.409385681152344e-05 [0:00:00.000034].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.479155Z[0m [[32m[1minfo     [0m] [1mWaiting for 599.9834852218628 seconds [0:09:59.983485], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.495506Z[0m [success  [0m] [1mFunction: lambda: self.instance succeeded after 4.220008850097656e-05 [0:00:00.000042].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.495715Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: bbr_gateway_namespace[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.509848Z[0m [[32m[1minfo     [0m] [1mExecuting class fixture: bbr_envoy_filter_config_patches[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.548651Z[0m [[32m[1minfo     [0m] [1mkind: EnvoyFilter api version: networking.istio.io/v1alpha3[0m [[0m[1m[34mocp_resources.resource[0m][0m
PASSED
_ 1 of 10 completed, 1 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _

tests/model_serving/maas_billing/body_base_routing/pre_auth_model_header/test_bbr_envoy_filter.py::TestBBREnvoyFilter::test_bbr_post_auth_processing_deployment_ready [1m--------------------------------------------------------- CALL ---------------------------------------------------------[0m
[2m2026-11-01T13:53:44.578285Z[0m [[32m[1minfo     [0m] [1mEnvoyFilter 'openshift-ingress/payload-processing' has pre-auth and post-auth stages: ['envoy.filters.http.ext_proc.ipp-pre', 'envoy.filters.http.ext_proc.ipp'][0m [[0m[1m[34mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m][0m [36mname[0m=[35mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m
[1m
TEST: TestBBREnvoyFilter.test_bbr_envoy_filter_has_pre_and_post_auth_stages STATUS: [0;32mPASSED[0m[0m
[1m------------------------------------------------------- TEARDOWN -------------------------------------------------------[0m
[1m
------------------------------------ test_bbr_post_auth_processing_deployment_ready ------------------------------------[0m
[1m-------------------------------------------------------- SETUP --------------------------------------------------------[0m
[1m--------------------------------------------------------- CALL ---------------------------------------------------------[0m
[2m2026-11-01T13:53:44.585517Z[0m [[32m[1minfo     [0m] [1mkind: Deployment api version: apps/v1[0m [[0m[1m[34mocp_resources.resource[0m][0m
PASSED
_ 2 of 10 completed, 2 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _

tests/model_serving/maas_billing/body_base_routing/pre_auth_model_header/test_bbr_post_auth_setup.py::TestBBRPostAuthSetup::test_bbr_post_processing_deployment_ready [2m2026-11-01T13:53:44.642280Z[0m [[32m[1minfo     [0m] [1mDeployment 'openshift-ingress/payload-processing' is ready (1/1 replicas)[0m [[0m[1m[34mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m][0m [36mname[0m=[35mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m
[1m
TEST: TestBBREnvoyFilter.test_bbr_post_auth_processing_deployment_ready STATUS: [0;32mPASSED[0m[0m
[1m------------------------------------------------------- TEARDOWN -------------------------------------------------------[0m
[1m
-------------------------------------- test_bbr_post_processing_deployment_ready --------------------------------------[0m
[1m-------------------------------------------------------- SETUP --------------------------------------------------------[0m
[1m--------------------------------------------------------- CALL ---------------------------------------------------------[0m
PASSED
_ 3 of 10 completed, 3 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _

tests/model_serving/maas_billing/body_base_routing/pre_auth_model_header/test_bbr_post_auth_setup.py::TestBBRPostAuthSetup::test_bbr_post_processing_service_exposes_port_9004 [2m2026-11-01T13:53:44.704017Z[0m [[32m[1minfo     [0m] [1mDeployment 'openshift-ingress/payload-processing' is ready (1/1 replicas)[0m [[0m[1m[34mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m][0m [36mname[0m=[35mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m
[1m
TEST: TestBBRPostAuthSetup.test_bbr_post_processing_deployment_ready STATUS: [0;32mPASSED[0m[0m
[1m------------------------------------------------------- TEARDOWN -------------------------------------------------------[0m
[1m
---------------------------------- test_bbr_post_processing_service_exposes_port_9004 ----------------------------------[0m
[1m-------------------------------------------------------- SETUP --------------------------------------------------------[0m
[1m--------------------------------------------------------- CALL ---------------------------------------------------------[0m
PASSED
_ 4 of 10 completed, 4 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _

tests/model_serving/maas_billing/body_base_routing/pre_auth_model_header/test_bbr_post_auth_setup.py::TestBBRPostAuthSetup::test_bbr_post_processing_destination_rule_exists [2m2026-11-01T13:53:44.734822Z[0m [[32m[1minfo     [0m] [1mService 'openshift-ingress/payload-processing' exposes gRPC port 9004[0m [[0m[1m[34mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m][0m [36mname[0m=[35mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m
[1m
TEST: TestBBRPostAuthSetup.test_bbr_post_processing_service_exposes_port_9004 STATUS: [0;32mPASSED[0m[0m
[1m------------------------------------------------------- TEARDOWN -------------------------------------------------------[0m
[1m
----------------------------------- test_bbr_post_processing_destination_rule_exists -----------------------------------[0m
[1m-------------------------------------------------------- SETUP --------------------------------------------------------[0m
[1m--------------------------------------------------------- CALL ---------------------------------------------------------[0m
[2m2026-11-01T13:53:44.738070Z[0m [[32m[1minfo     [0m] [1mkind: DestinationRule api version: networking.istio.io/v1[0m [[0m[1m[34mocp_resources.resource[0m][0m
PASSED
_ 5 of 10 completed, 5 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _

tests/model_serving/maas_billing/body_base_routing/pre_auth_model_header/test_bbr_post_auth_setup.py::TestBBRPostAuthSetup::test_bbr_plugins_configmap_has_expected_plugins [2m2026-11-01T13:53:44.750809Z[0m [[32m[1minfo     [0m] [1mDestinationRule 'openshift-ingress/payload-processing' exists[0m [[0m[1m[34mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m][0m [36mname[0m=[35mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m
[1m
TEST: TestBBRPostAuthSetup.test_bbr_post_processing_destination_rule_exists STATUS: [0;32mPASSED[0m[0m
[1m------------------------------------------------------- TEARDOWN -------------------------------------------------------[0m
[1m
----------------------------------- test_bbr_plugins_configmap_has_expected_plugins -----------------------------------[0m
[1m-------------------------------------------------------- SETUP --------------------------------------------------------[0m
[1m--------------------------------------------------------- CALL ---------------------------------------------------------[0m
[2m2026-11-01T13:53:44.753931Z[0m [[32m[1minfo     [0m] [1mWaiting for 300 seconds [0:05:00], retry every 5 seconds. (Function: tests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils._fetch_bbr_plugins_configmap_data_if_ready  Kwargs: {'admin_client': <kubernetes.dynamic.client.DynamicClient object at 0x7fea4c9e6350>, 'gateway_namespace': 'openshift-ingress'})[0m [[0m[1m[34mtimeout_sampler[0m][0m
PASSED
_ 6 of 10 completed, 6 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _

tests/model_serving/maas_billing/body_base_routing/pre_auth_model_header/test_bbr_setup.py::TestBBRSetup::test_bbr_pre_processing_service_exposes_port_9004 [2m2026-11-01T13:53:44.780895Z[0m [[32m[1minfo     [0m] [1mConfigMap 'openshift-ingress/payload-processing-plugins' has post-auth and pre-auth plugin keys[0m [[0m[1m[34mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m][0m [36mname[0m=[35mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m
[2m2026-11-01T13:53:44.785522Z[0m [success  [0m] [1mFunction: tests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils._fetch_bbr_plugins_configmap_data_if_ready succeeded after 5.1975250244140625e-05 [0:00:00.000052].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:44.782269Z[0m [[32m[1minfo     [0m] [1mPost-auth plugins verified: ['model-provider-resolver', 'api-translation', 'apikey-injection'][0m [[0m[1m[34mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m][0m [36mname[0m=[35mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m
[2m2026-11-01T13:53:44.783099Z[0m [[32m[1minfo     [0m] [1mPre-auth plugin 'body-field-to-header' correctly maps 'model' → 'X-Gateway-Model-Name'[0m [[0m[1m[34mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m][0m [36mname[0m=[35mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m
[1m
TEST: TestBBRPostAuthSetup.test_bbr_plugins_configmap_has_expected_plugins STATUS: [0;32mPASSED[0m[0m
[1m------------------------------------------------------- TEARDOWN -------------------------------------------------------[0m
[1m
---------------------------------- test_bbr_pre_processing_service_exposes_port_9004 ----------------------------------[0m
[1m-------------------------------------------------------- SETUP --------------------------------------------------------[0m
[1m--------------------------------------------------------- CALL ---------------------------------------------------------[0m
PASSED[2m2026-11-01T13:53:44.814900Z[0m [[32m[1minfo     [0m] [1mService 'openshift-ingress/payload-pre-processing' exposes gRPC port 9004[0m [[0m[1m[34mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m][0m [36mname[0m=[35mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m

_ 7 of 10 completed, 7 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _

tests/model_serving/maas_billing/body_base_routing/pre_auth_model_header/test_bbr_setup.py::TestBBRSetup::test_bbr_pre_processing_destination_rule_exists [1m
TEST: TestBBRSetup.test_bbr_pre_processing_service_exposes_port_9004 STATUS: [0;32mPASSED[0m[0m
[1m------------------------------------------------------- TEARDOWN -------------------------------------------------------[0m
[1m
----------------------------------- test_bbr_pre_processing_destination_rule_exists -----------------------------------[0m
[1m-------------------------------------------------------- SETUP --------------------------------------------------------[0m
[1m--------------------------------------------------------- CALL ---------------------------------------------------------[0m
PASSED[2m2026-11-01T13:53:44.832061Z[0m [[32m[1minfo     [0m] [1mDestinationRule 'openshift-ingress/payload-pre-processing' exists[0m [[0m[1m[34mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m][0m [36mname[0m=[35mtests.model_serving.maas_billing.body_base_routing.pre_auth_model_header.utils[0m

_ 8 of 10 completed, 8 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _

tests/model_serving/maas_billing/multitenancy/aitenant/test_tenant_setup.py::TestAITenantTenantSetup::test_aitenant_crd_exists [1m
TEST: TestBBRSetup.test_bbr_pre_processing_destination_rule_exists STATUS: [0;32mPASSED[0m[0m
[1m------------------------------------------------------- TEARDOWN -------------------------------------------------------[0m
[1m
----------------------------------------------- test_aitenant_crd_exists -----------------------------------------------[0m
[1m-------------------------------------------------------- SETUP --------------------------------------------------------[0m
[1m--------------------------------------------------------- CALL ---------------------------------------------------------[0m
[2m2026-11-01T13:53:44.836680Z[0m [[32m[1minfo     [0m] [1mkind: CustomResourceDefinition api version: apiextensions.k8s.io/v1[0m [[0m[1m[34mocp_resources.resource[0m][0m
PASSED[2m2026-11-01T13:53:44.865009Z[0m [[32m[1minfo     [0m] [1mAITenant CRD 'aitenants.maas.opendatahub.io' exists[0m [[0m[1m[34m_pytest.compat[0m][0m [36mname[0m=[35mmultitenancy.aitenant.test_tenant_setup[0m
[1m
TEST: TestAITenantTenantSetup.test_aitenant_crd_exists STATUS: [0;32mPASSED[0m[0m
[1m------------------------------------------------------- TEARDOWN -------------------------------------------------------[0m

_ 9 of 10 completed, 9 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _

tests/model_serving/maas_billing/multitenancy/aitenant/test_tenant_setup.py::TestAITenantTenantSetup::test_aitenant_bootstrap_does_not_break_default_tenant [1m
-------------------------------- test_aitenant_bootstrap_does_not_break_default_tenant --------------------------------[0m
[1m-------------------------------------------------------- SETUP --------------------------------------------------------[0m
[2m2026-11-01T13:53:44.868669Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: aitenant_infra_namespace[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.884308Z[0m [[32m[1minfo     [0m] [1mExecuting session fixture: teardown_resources[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.884509Z[0m [[32m[1minfo     [0m] [1mExecuting function fixture: aitenant_test_params[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.884663Z[0m [[32m[1minfo     [0m] [1mExecuting function fixture: aitenant_bootstrap_gateway[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.884809Z[0m [[32m[1minfo     [0m] [1mCreate ConfigMap e2e-aigw-6ff0276a-config[0m [[0m[1m[34mocp_resources ConfigMap[0m][0m
[2m2026-11-01T13:53:44.884952Z[0m [[32m[1minfo     [0m] [1mPosting {'apiVersion': 'v1', 'kind': 'ConfigMap', 'metadata': {'name': 'e2e-aigw-6ff0276a-config', 'namespace': 'openshift-ingress'}, 'data': '*******'}[0m [[0m[1m[34mocp_resources ConfigMap[0m][0m
[2m2026-11-01T13:53:44.915705Z[0m [[32m[1minfo     [0m] [1mkind: Gateway api version: gateway.networking.k8s.io/v1[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:44.915877Z[0m [[32m[1minfo     [0m] [1mCreate Gateway e2e-aigw-6ff0276a[0m [[0m[1m[34mocp_resources Gateway[0m][0m
[2m2026-11-01T13:53:44.916040Z[0m [[32m[1minfo     [0m] [1mPosting {'apiVersion': 'gateway.networking.k8s.io/v1', 'kind': 'Gateway', 'metadata': {'name': 'e2e-aigw-6ff0276a', 'namespace': 'openshift-ingress'}, 'spec': {'gatewayClassName': 'openshift-default', 'listeners': [{'name': 'http', 'port': 80, 'protocol': 'HTTP', 'allowedRoutes': {'namespaces': {'from': 'All'}}}, {'name': 'https', 'port': 443, 'protocol': 'HTTPS', 'allowedRoutes': {'namespaces': {'from': 'All'}}, 'tls': {'mode': 'Terminate', 'certificateRefs': [{'group': '', 'kind': 'Secret', 'name': 'data-science-gateway-service-tls'}]}}], 'infrastructure': {'parametersRef': {'group': '', 'kind': 'ConfigMap', 'name': 'e2e-aigw-6ff0276a-config'}}}}[0m [[0m[1m[34mocp_resources Gateway[0m][0m
[2m2026-11-01T13:53:44.946891Z[0m [[32m[1minfo     [0m] [1mExecuting function fixture: aitenant[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:44.973003Z[0m [[32m[1minfo     [0m] [1mkind: AITenant api version: maas.opendatahub.io/v1alpha1[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:44.973247Z[0m [[32m[1minfo     [0m] [1mCreate AITenant e2e-aigw-6ff0276a[0m [[0m[1m[34mocp_resources AITenant[0m][0m
[2m2026-11-01T13:53:44.973392Z[0m [[32m[1minfo     [0m] [1mPosting {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'name': 'e2e-aigw-6ff0276a', 'namespace': 'ai-tenants'}, 'spec': {'gateway': {'name': 'e2e-aigw-6ff0276a'}}}[0m [[0m[1m[34mocp_resources AITenant[0m][0m
[2m2026-11-01T13:53:45.168158Z[0m [[32m[1minfo     [0m] [1mWait until AITenant e2e-aigw-6ff0276a is created[0m [[0m[1m[34mocp_resources AITenant[0m][0m
[2m2026-11-01T13:53:45.168372Z[0m [[32m[1minfo     [0m] [1mWaiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:45.182485Z[0m [success  [0m] [1mFunction: lambda: self.exists succeeded after 5.91278076171875e-05 [0:00:00.000059].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:45.182657Z[0m [[32m[1minfo     [0m] [1mExecuting function fixture: ready_aitenant[0m [[0m[1m[34mconftest[0m][0m
[2m2026-11-01T13:53:45.210490Z[0m [[32m[1minfo     [0m] [1mWait for AITenant/e2e-aigw-6ff0276a's 'Ready' condition to be 'True'[0m [[0m[1m[34mocp_resources AITenant[0m][0m
[2m2026-11-01T13:53:45.210678Z[0m [[32m[1minfo     [0m] [1mWait until AITenant e2e-aigw-6ff0276a is created[0m [[0m[1m[34mocp_resources AITenant[0m][0m
[2m2026-11-01T13:53:45.210813Z[0m [[32m[1minfo     [0m] [1mWaiting for 300 seconds [0:05:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:45.224990Z[0m [success  [0m] [1mFunction: lambda: self.exists succeeded after 5.888938903808594e-05 [0:00:00.000059].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:45.225229Z[0m [[32m[1minfo     [0m] [1mWaiting for 299.9852910041809 seconds [0:04:59.985291], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:46.254697Z[0m [success  [0m] [1mFunction: lambda: self.instance succeeded after 1.0142533779144287 [0:00:01.014253].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:46.269225Z[0m [[32m[1minfo     [0m] [1mExecuting function fixture: aitenant_for_test[0m [[0m[1m[34mconftest[0m][0m
[1m--------------------------------------------------------- CALL ---------------------------------------------------------[0m
[2m2026-11-01T13:53:46.298301Z[0m [[32m[1minfo     [0m] [1mWait for AITenant/e2e-aigw-6ff0276a's 'Ready' condition to be 'True'[0m [[0m[1m[34mocp_resources AITenant[0m][0m
[2m2026-11-01T13:53:46.298475Z[0m [[32m[1minfo     [0m] [1mWait until AITenant e2e-aigw-6ff0276a is created[0m [[0m[1m[34mocp_resources AITenant[0m][0m
[2m2026-11-01T13:53:46.298617Z[0m [[32m[1minfo     [0m] [1mWaiting for 300 seconds [0:05:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:46.312498Z[0m [success  [0m] [1mFunction: lambda: self.exists succeeded after 5.364418029785156e-05 [0:00:00.000054].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:46.312663Z[0m [[32m[1minfo     [0m] [1mWaiting for 299.98565578460693 seconds [0:04:59.985656], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:46.327075Z[0m [success  [0m] [1mFunction: lambda: self.instance succeeded after 4.649162292480469e-05 [0:00:00.000046].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:46.399575Z[0m [[33m[1mwarning  [0m] [1mCouldn't find MaasTenantConfig in maas.opendatahub.io api group[0m [[0m[1m[34mocp_resources.resource[0m][0m
FAILED[1m
TEST: TestAITenantTenantSetup.test_aitenant_bootstrap_does_not_break_default_tenant STATUS: [0;31mFAILED[0m[0m
[2m2026-11-01T13:53:46.405221Z[0m [[31m[1merror    [0m] [1mTraceback (most recent call last):
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call
    result: TResult | None = func()
                             ~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in <lambda>
    lambda: runtest_hook(item=item, **kwds),
            ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
    return result.get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result
    raise exc.with_traceback(tb)
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
    res = yield
          ^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call
    item.runtest()
    ~~~~~~~~~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
    return result.get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result
    raise exc.with_traceback(tb)
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
    res = yield
          ^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/odh/opendatahub-tests/tests/model_serving/maas_billing/multitenancy/aitenant/test_tenant_setup.py", line 54, in test_aitenant_bootstrap_does_not_break_default_tenant
    verify_default_maas_tenant_unaffected(admin_client=admin_client)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/tests/model_serving/maas_billing/multitenancy/aitenant/utils.py", line 761, in verify_default_maas_tenant_unaffected
    default_maas_tenant_config = MaasTenantConfig(
        client=admin_client,
        name=AIGATEWAY_BOOTSTRAPPED_TENANT_NAME,
        namespace=MAAS_SUBSCRIPTION_NAMESPACE,
    )
  File "/home/odh/opendatahub-tests/utilities/resources/maastenantconfig.py", line 31, in __init__
    super().__init__(**kwargs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 1637, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
        name=name,
        ^^^^^^^^^^
    ...<4 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 713, in __init__
    self._set_api_version()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 898, in _set_api_version
    self.api_version = _get_api_version(client=self.client, api_group=self.api_group, kind=self.kind)
                       ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 87, in _get_api_version
    raise NotImplementedError(log)
NotImplementedError: Couldn't find MaasTenantConfig in maas.opendatahub.io api group[0m [[0m[1m[34mconftest[0m][0m
[1m------------------------------------------------------- TEARDOWN -------------------------------------------------------[0m
[2m2026-11-01T13:53:46.405515Z[0m [[32m[1minfo     [0m] [1mDelete AITenant e2e-aigw-6ff0276a[0m [[0m[1m[34mocp_resources AITenant[0m][0m
[2m2026-11-01T13:53:46.435681Z[0m [[32m[1minfo     [0m] [1mDeleting {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'creationTimestamp': '2026-11-01T13:53:44Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'managedFields': [{'apiVersion': 'maas.opendatahub.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'.': {}, 'f:gateway': {'.': {}, 'f:name': {}}}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-11-01T13:53:44Z'}, {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"maas.opendatahub.io/aitenant-cleanup"': {}}}}, 'manager': 'manager', 'operation': 'Update', 'time': '2026-11-01T13:53:45Z'}, {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'.': {}, 'f:conditions': {}, 'f:gatewayRef': {'.': {}, 'f:name': {}, 'f:namespace': {}}, 'f:phase': {}, 'f:tenantNamespace': {}}}, 'manager': 'manager', 'operation': 'Update', 'subresource': 'status', 'time': '2026-11-01T13:53:46Z'}], 'name': 'e2e-aigw-6ff0276a', 'namespace': 'ai-tenants', 'resourceVersion': '110807', 'uid': '9af6df0f-141e-4d89-af25-d4601425b771'}, 'spec': {'gateway': {'name': 'e2e-aigw-6ff0276a'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-11-01T13:53:46Z', 'message': 'AITenant bootstrap resources are reconciled', 'observedGeneration': 1, 'reason': 'Reconciled', 'status': 'True', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-aigw-6ff0276a', 'namespace': 'openshift-ingress'}, 'phase': 'Active', 'tenantNamespace': 'ai-tenant-e2e-aigw-6ff0276a'}}[0m [[0m[1m[34mocp_resources AITenant[0m][0m
[2m2026-11-01T13:53:46.453576Z[0m [[32m[1minfo     [0m] [1mWait until AITenant e2e-aigw-6ff0276a is deleted[0m [[0m[1m[34mocp_resources AITenant[0m][0m
[2m2026-11-01T13:53:46.453726Z[0m [[32m[1minfo     [0m] [1mWaiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:47.484499Z[0m [success  [0m] [1mFunction: lambda: self.exists succeeded after 1.014291524887085 [0:00:01.014292].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:47.484681Z[0m [[32m[1minfo     [0m] [1mDelete Gateway e2e-aigw-6ff0276a[0m [[0m[1m[34mocp_resources Gateway[0m][0m
[2m2026-11-01T13:53:47.536776Z[0m [[32m[1minfo     [0m] [1mDeleting {'apiVersion': 'gateway.networking.k8s.io/v1', 'kind': 'Gateway', 'metadata': {'creationTimestamp': '2026-11-01T13:53:44Z', 'generation': 1, 'labels': {'istio.io/rev': 'openshift-gateway'}, 'managedFields': [{'apiVersion': 'gateway.networking.k8s.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:gateway.istio.io/controller-version': {}}}}, 'manager': 'openshift.io/gateway-controller/v1', 'operation': 'Apply', 'subresource': 'status', 'time': '2026-11-01T13:53:44Z'}, {'apiVersion': 'gateway.networking.k8s.io/v1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {}}, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': '2026-11-01T13:53:44Z'}, {'apiVersion': 'gateway.networking.k8s.io/v1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:istio.io/rev': {}}}}, 'manager': 'ingress-operator', 'operation': 'Update', 'time': '2026-11-01T13:53:44Z'}, {'apiVersion': 'gateway.networking.k8s.io/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:status': {'f:addresses': {}, 'f:conditions': {'k:{"type":"Accepted"}': {'f:lastTransitionTime': {}, 'f:message': {}, 'f:observedGeneration': {}, 'f:reason': {}, 'f:status': {}}, 'k:{"type":"Programmed"}': {'f:lastTransitionTime': {}, 'f:message': {}, 'f:observedGeneration': {}, 'f:reason': {}, 'f:status': {}}}, 'f:listeners': {'.': {}, 'k:{"name":"http"}': {'.': {}, 'f:attachedRoutes': {}, 'f:conditions': {'.': {}, 'k:{"type":"Accepted"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:message': {}, 'f:observedGeneration': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"Conflicted"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:message': {}, 'f:observedGeneration': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"Programmed"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:message': {}, 'f:observedGeneration': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"ResolvedRefs"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:message': {}, 'f:observedGeneration': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}}, 'f:name': {}, 'f:supportedKinds': {}}, 'k:{"name":"https"}': {'.': {}, 'f:attachedRoutes': {}, 'f:conditions': {'.': {}, 'k:{"type":"Accepted"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:message': {}, 'f:observedGeneration': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"Conflicted"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:message': {}, 'f:observedGeneration': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"Programmed"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:message': {}, 'f:observedGeneration': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}, 'k:{"type":"ResolvedRefs"}': {'.': {}, 'f:lastTransitionTime': {}, 'f:message': {}, 'f:observedGeneration': {}, 'f:reason': {}, 'f:status': {}, 'f:type': {}}}, 'f:name': {}, 'f:supportedKinds': {}}}}}, 'manager': 'pilot-discovery', 'operation': 'Update', 'subresource': 'status', 'time': '2026-11-01T13:53:47Z'}], 'name': 'e2e-aigw-6ff0276a', 'namespace': 'openshift-ingress', 'resourceVersion': '110852', 'uid': 'fcc4562d-4151-4793-8989-4928a32b41e6'}, 'spec': {'gatewayClassName': 'openshift-default', 'infrastructure': {'parametersRef': {'group': '', 'kind': 'ConfigMap', 'name': 'e2e-aigw-6ff0276a-config'}}, 'listeners': [{'allowedRoutes': {'namespaces': {'from': 'All'}}, 'name': 'http', 'port': 80, 'protocol': 'HTTP'}, {'allowedRoutes': {'namespaces': {'from': 'All'}}, 'name': 'https', 'port': 443, 'protocol': 'HTTPS', 'tls': {'certificateRefs': [{'group': '', 'kind': 'Secret', 'name': 'data-science-gateway-service-tls'}], 'mode': 'Terminate'}}]}, 'status': {'addresses': [{'type': 'Hostname', 'value': 'e2e-aigw-6ff0276a-openshift-default.openshift-ingress.svc.cluster.local'}], 'conditions': [{'lastTransitionTime': '2026-11-01T13:53:45Z', 'message': 'Resource accepted', 'observedGeneration': 1, 'reason': 'Accepted', 'status': 'True', 'type': 'Accepted'}, {'lastTransitionTime': '2026-11-01T13:53:46Z', 'message': 'Resource programmed, assigned to service(s) e2e-aigw-6ff0276a-openshift-default.openshift-ingress.svc.cluster.local:443 and e2e-aigw-6ff0276a-openshift-default.openshift-ingress.svc.cluster.local:80', 'observedGeneration': 1, 'reason': 'Programmed', 'status': 'True', 'type': 'Programmed'}], 'listeners': [{'attachedRoutes': 0, 'conditions': [{'lastTransitionTime': '2026-11-01T13:53:45Z', 'message': 'No errors found', 'observedGeneration': 1, 'reason': 'Accepted', 'status': 'True', 'type': 'Accepted'}, {'lastTransitionTime': '2026-11-01T13:53:45Z', 'message': 'No errors found', 'observedGeneration': 1, 'reason': 'NoConflicts', 'status': 'False', 'type': 'Conflicted'}, {'lastTransitionTime': '2026-11-01T13:53:45Z', 'message': 'No errors found', 'observedGeneration': 1, 'reason': 'Programmed', 'status': 'True', 'type': 'Programmed'}, {'lastTransitionTime': '2026-11-01T13:53:45Z', 'message': 'No errors found', 'observedGeneration': 1, 'reason': 'ResolvedRefs', 'status': 'True', 'type': 'ResolvedRefs'}], 'name': 'http', 'supportedKinds': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute'}, {'group': 'gateway.networking.k8s.io', 'kind': 'GRPCRoute'}]}, {'attachedRoutes': 0, 'conditions': [{'lastTransitionTime': '2026-11-01T13:53:45Z', 'message': 'No errors found', 'observedGeneration': 1, 'reason': 'Accepted', 'status': 'True', 'type': 'Accepted'}, {'lastTransitionTime': '2026-11-01T13:53:45Z', 'message': 'No errors found', 'observedGeneration': 1, 'reason': 'NoConflicts', 'status': 'False', 'type': 'Conflicted'}, {'lastTransitionTime': '2026-11-01T13:53:45Z', 'message': 'No errors found', 'observedGeneration': 1, 'reason': 'Programmed', 'status': 'True', 'type': 'Programmed'}, {'lastTransitionTime': '2026-11-01T13:53:45Z', 'message': 'No errors found', 'observedGeneration': 1, 'reason': 'ResolvedRefs', 'status': 'True', 'type': 'ResolvedRefs'}], 'name': 'https', 'supportedKinds': [{'group': 'gateway.networking.k8s.io', 'kind': 'HTTPRoute'}, {'group': 'gateway.networking.k8s.io', 'kind': 'GRPCRoute'}]}]}}[0m [[0m[1m[34mocp_resources Gateway[0m][0m
[2m2026-11-01T13:53:47.554397Z[0m [[32m[1minfo     [0m] [1mWait until Gateway e2e-aigw-6ff0276a is deleted[0m [[0m[1m[34mocp_resources Gateway[0m][0m
[2m2026-11-01T13:53:47.554568Z[0m [[32m[1minfo     [0m] [1mWaiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:47.573512Z[0m [success  [0m] [1mFunction: lambda: self.exists succeeded after 6.031990051269531e-05 [0:00:00.000060].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:47.573737Z[0m [[32m[1minfo     [0m] [1mDelete ConfigMap e2e-aigw-6ff0276a-config[0m [[0m[1m[34mocp_resources ConfigMap[0m][0m
[2m2026-11-01T13:53:48.568006Z[0m [[32m[1minfo     [0m] [1mDeleting {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'e2e-aigw-6ff0276a-config', 'namespace': 'openshift-ingress', 'uid': '1d365927-f712-4ec3-a63c-b41466e7fcd9', 'resourceVersion': '110670', 'creationTimestamp': '2026-11-01T13:53:44Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-11-01T13:53:44Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:data': {'.': {}, 'f:service': {}}}}]}, 'data': '*******'}[0m [[0m[1m[34mocp_resources ConfigMap[0m][0m
[2m2026-11-01T13:53:48.584020Z[0m [[32m[1minfo     [0m] [1mWait until ConfigMap e2e-aigw-6ff0276a-config is deleted[0m [[0m[1m[34mocp_resources ConfigMap[0m][0m
[2m2026-11-01T13:53:48.584411Z[0m [[32m[1minfo     [0m] [1mWaiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:48.599855Z[0m [success  [0m] [1mFunction: lambda: self.exists succeeded after 5.793571472167969e-05 [0:00:00.000058].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:48.600016Z[0m [[32m[1minfo     [0m] [1mResourceEdits: Restoring data for resource DataScienceCluster default-dsc[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:48.600153Z[0m [[32m[1minfo     [0m] [1mUpdate DataScienceCluster default-dsc:
{'spec': {'components': {'kserve': {'modelsAsService': {}}}}, 'metadata': {'name': 'default-dsc'}}[0m [[0m[1m[34mocp_resources DataScienceCluster[0m][0m
[2m2026-11-01T13:53:48.887163Z[0m [[32m[1minfo     [0m] [1mResourceEdit: Backing up old data[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:48.903915Z[0m [[32m[1minfo     [0m] [1mResourceEdits: Updating data for resource DataScienceCluster default-dsc[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:48.904090Z[0m [[32m[1minfo     [0m] [1mUpdate DataScienceCluster default-dsc:
{'spec': {'components': {'kserve': {'modelsAsService': {'managementState': 'Managed'}}}}, 'metadata': {'name': 'default-dsc'}}[0m [[0m[1m[34mocp_resources DataScienceCluster[0m][0m
[2m2026-11-01T13:53:49.039800Z[0m [[32m[1minfo     [0m] [1mResourceEdits: Restoring data for resource DataScienceCluster default-dsc[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:49.039973Z[0m [[32m[1minfo     [0m] [1mUpdate DataScienceCluster default-dsc:
{'spec': {'components': {'kserve': {'modelsAsService': {}}}}, 'metadata': {'name': 'default-dsc'}}[0m [[0m[1m[34mocp_resources DataScienceCluster[0m][0m
[2m2026-11-01T13:53:49.178660Z[0m [[32m[1minfo     [0m] [1mWait for DataScienceCluster/default-dsc's 'Ready' condition to be 'True'[0m [[0m[1m[34mocp_resources DataScienceCluster[0m][0m
[2m2026-11-01T13:53:49.178841Z[0m [[32m[1minfo     [0m] [1mWait until DataScienceCluster default-dsc is created[0m [[0m[1m[34mocp_resources DataScienceCluster[0m][0m
[2m2026-11-01T13:53:49.178985Z[0m [[32m[1minfo     [0m] [1mWaiting for 600 seconds [0:10:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:49.195244Z[0m [success  [0m] [1mFunction: lambda: self.exists succeeded after 5.7220458984375e-05 [0:00:00.000057].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:49.195408Z[0m [[32m[1minfo     [0m] [1mWaiting for 599.9831132888794 seconds [0:09:59.983113], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_condition.lambda: self.instance)[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:49.212512Z[0m [success  [0m] [1mFunction: lambda: self.instance succeeded after 4.029273986816406e-05 [0:00:00.000040].[0m [[0m[1m[34mtimeout_sampler[0m][0m
[2m2026-11-01T13:53:49.212676Z[0m [[32m[1minfo     [0m] [1mResourceEdits: Restoring data for resource Gateway maas-default-gateway[0m [[0m[1m[34mocp_resources.resource[0m][0m
[2m2026-11-01T13:53:49.212832Z[0m [[32m[1minfo     [0m] [1mUpdate Gateway maas-default-gateway:
{'metadata': {'annotations': {}, 'name': 'maas-default-gateway'}}[0m [[0m[1m[34mocp_resources Gateway[0m][0m

_ TestAITenantTenantSetup.test_aitenant_bootstrap_does_not_break_default_tenant _
Traceback (most recent call last):
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call
    result: TResult | None = func()
                             ~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in <lambda>
    lambda: runtest_hook(item=item, **kwds),
            ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
    return result.get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result
    raise exc.with_traceback(tb)
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
    res = yield
          ^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call
    item.runtest()
    ~~~~~~~~~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
    return result.get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result
    raise exc.with_traceback(tb)
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
    res = yield
          ^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/odh/opendatahub-tests/tests/model_serving/maas_billing/multitenancy/aitenant/test_tenant_setup.py", line 54, in test_aitenant_bootstrap_does_not_break_default_tenant
    verify_default_maas_tenant_unaffected(admin_client=admin_client)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/tests/model_serving/maas_billing/multitenancy/aitenant/utils.py", line 761, in verify_default_maas_tenant_unaffected
    default_maas_tenant_config = MaasTenantConfig(
        client=admin_client,
        name=AIGATEWAY_BOOTSTRAPPED_TENANT_NAME,
        namespace=MAAS_SUBSCRIPTION_NAMESPACE,
    )
  File "/home/odh/opendatahub-tests/utilities/resources/maastenantconfig.py", line 31, in __init__
    super().__init__(**kwargs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 1637, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
        name=name,
        ^^^^^^^^^^
    ...<4 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 713, in __init__
    self._set_api_version()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 898, in _set_api_version
    self.api_version = _get_api_version(client=self.client, api_group=self.api_group, kind=self.kind)
                       ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 87, in _get_api_version
    raise NotImplementedError(log)
NotImplementedError: Couldn't find MaasTenantConfig in maas.opendatahub.io api group
_ 10 of 10 completed, 9 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _
================= 1 failed, 9 passed, 208 deselected in 6.98s ==================

=================================== FAILURES ===================================
_ TestAITenantTenantSetup.test_aitenant_bootstrap_does_not_break_default_tenant _
Traceback (most recent call last):
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call
    result: TResult | None = func()
                             ~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in <lambda>
    lambda: runtest_hook(item=item, **kwds),
            ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
    return result.get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result
    raise exc.with_traceback(tb)
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
    res = yield
          ^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call
    item.runtest()
    ~~~~~~~~~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
    return result.get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result
    raise exc.with_traceback(tb)
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
    res = yield
          ^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/odh/opendatahub-tests/tests/model_serving/maas_billing/multitenancy/aitenant/test_tenant_setup.py", line 54, in test_aitenant_bootstrap_does_not_break_default_tenant
    verify_default_maas_tenant_unaffected(admin_client=admin_client)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/tests/model_serving/maas_billing/multitenancy/aitenant/utils.py", line 761, in verify_default_maas_tenant_unaffected
    default_maas_tenant_config = MaasTenantConfig(
        client=admin_client,
        name=AIGATEWAY_BOOTSTRAPPED_TENANT_NAME,
        namespace=MAAS_SUBSCRIPTION_NAMESPACE,
    )
  File "/home/odh/opendatahub-tests/utilities/resources/maastenantconfig.py", line 31, in __init__
    super().__init__(**kwargs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 1637, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
        name=name,
        ^^^^^^^^^^
    ...<4 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 713, in __init__
    self._set_api_version()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 898, in _set_api_version
    self.api_version = _get_api_version(client=self.client, api_group=self.api_group, kind=self.kind)
                       ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 87, in _get_api_version
    raise NotImplementedError(log)
NotImplementedError: Couldn't find MaasTenantConfig in maas.opendatahub.io api group
- generated xml file: /workspace/tests-shared/tests-payload/results/maas_billing-smoke.xml -
=========================== short test summary info ============================
FAILED tests/model_serving/maas_billing/multitenancy/aitenant/test_tenant_setup.py::TestAITenantTenantSetup::test_aitenant_bootstrap_does_not_break_default_tenant - NotImplementedError: Couldn't find MaasTenantConfig in maas.opendatahub.io api group
================= 1 failed, 9 passed, 208 deselected in 6.98s ==================