--- BVT pytest attempt 2026-06-03T14:51:55.774476+00:00 (/.local/bin/uv run pytest...) --- 2026-06-03T14:51:56.995837Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] 2026-06-03T14:51:57.007298Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.007536Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.008271Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.008455Z [info ] PRAGMA main.table_info("OpenDataHubTestTable") [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.008719Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.008898Z [info ] [raw sql] ()  [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.009053Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.009250Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.012873Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-03T14:51:57.021860Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-06-03T14:51:57.065350Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-03T14:51:57.073492Z [info ] Running model registry tests against default namespace [conftest] 2026-06-03T14:51:57.073767Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] ============================= test session starts ============================== platform linux -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.14.5', 'Platform': 'Linux-5.14.0-427.115.1.el9_4.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.0.3', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.13.0', 'asyncio': '1.3.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.3.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '5.1.0'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.13.0, asyncio-1.3.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.3.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-5.1.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... collected 11 items tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::TestMaaSApiComponentHealth::test_maas_management_state  ---------------------------------------------- test_maas_management_state ---------------------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-03T14:51:57.126486Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() started [sqlalchemy.orm.mapper.Mapper] 2026-06-03T14:51:57.126734Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop id [sqlalchemy.orm.mapper.Mapper] 2026-06-03T14:51:57.126922Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop test_name [sqlalchemy.orm.mapper.Mapper] 2026-06-03T14:51:57.127091Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) initialize prop start_time [sqlalchemy.orm.mapper.Mapper] 2026-06-03T14:51:57.127243Z [info ] (OpenDataHubTestTable|OpenDataHubTestTable) _post_configure_properties() complete [sqlalchemy.orm.mapper.Mapper] 2026-06-03T14:51:57.127908Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.128069Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.128214Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.129322Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.129790Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_management_state', 1780498317) [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.129975Z [info ] [generated in 0.00013s] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_management_state', 1780498317) [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.130125Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.139089Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.139712Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-06-03T14:51:57.140048Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-06-03T14:51:57.140189Z [info ] Executing session fixture: admin_client [conftest] 2026-06-03T14:51:57.140329Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-03T14:51:57.140456Z [info ] Executing session fixture: dsc_resource [conftest] 2026-06-03T14:51:57.140598Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-06-03T14:51:57.171826Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-06-03T14:51:57.172042Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-06-03T14:51:57.173229Z [info ] Executing session fixture: os_path_environment [conftest] 2026-06-03T14:51:57.173436Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-06-03T14:51:57.173672Z [info ] Executing session fixture: bin_directory [conftest] 2026-06-03T14:51:57.174756Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-06-03T14:51:57.174951Z [info ] kind: ConsoleCLIDownload api version: console.openshift.io/v1 [ocp_resources.resource] 2026-06-03T14:51:57.181353Z [info ] All oc console cli download links: ['https://downloads-openshift-console.apps.9a96c5ff-548b-4b93-9f8e-a6229be599cf.prod.konfluxeaas.com/amd64/linux/oc.tar'] [utilities.infra] name=utilities.infra 2026-06-03T14:51:57.181450Z [info ] Downloading archive using: url=https://downloads-openshift-console.apps.9a96c5ff-548b-4b93-9f8e-a6229be599cf.prod.konfluxeaas.com/amd64/linux/oc.tar [utilities.infra] name=utilities.infra 2026-06-03T14:51:57.775310Z [info ] Extract the downloaded archive. [utilities.infra] name=utilities.infra 2026-06-03T14:51:57.866213Z [info ] Downloaded file: ['oc']  [utilities.infra] name=utilities.infra 2026-06-03T14:51:57.887806Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-06-03T14:51:57.886101Z [info ] OC binary path: /tmp/pytest-AVXJTx7WcZfDkgL8goPmsD/bin0/oc [tests.conftest] name=tests.conftest 2026-06-03T14:51:57.886164Z [info ] Adding /tmp/pytest-AVXJTx7WcZfDkgL8goPmsD/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-06-03T14:51:57.888266Z [info ] Executing session fixture: nodes [conftest] 2026-06-03T14:51:57.898358Z [info ] Executing session fixture: dsci_resource [conftest] 2026-06-03T14:51:57.898525Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-06-03T14:51:57.903249Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-06-03T14:51:57.903425Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-06-03T14:51:57.903581Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-06-03T14:51:57.902359Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-06-03T14:51:57.903931Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-06-03T14:51:57.914518Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-06-03T14:51:57.926218Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsci_status_ready Kwargs: {'dsci_resource': }) [timeout_sampler] 2026-06-03T14:51:57.925479Z [info ] Wait for DSCI default-dsci to be in Ready status. [utilities.infra] name=utilities.infra 2026-06-03T14:51:57.926557Z [info ] Get DSCInitialization default-dsci status [ocp_resources DSCInitialization] 2026-06-03T14:51:57.929657Z [info ] Elapsed time: 7.486343383789062e-05 [0:00:00.000075] [timeout_sampler] 2026-06-03T14:51:57.929829Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': }) [timeout_sampler] 2026-06-03T14:51:57.928972Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-06-03T14:51:57.930112Z [info ] Get DataScienceCluster default-dsc status [ocp_resources DataScienceCluster] 2026-06-03T14:51:57.935201Z [info ] Elapsed time: 3.3855438232421875e-05 [0:00:00.000034] [timeout_sampler] 2026-06-03T14:51:57.935374Z [info ] Executing session fixture: autouse_fixtures [conftest] 2026-06-03T14:51:57.935528Z [info ] Executing session fixture: maas_gateway_api_hostname [conftest] 2026-06-03T14:51:57.935712Z [info ] kind: Ingress api version: config.openshift.io/v1 [ocp_resources.resource] 2026-06-03T14:51:57.942435Z [info ] Executing session fixture: maas_gateway_api [conftest] 2026-06-03T14:51:57.942605Z [info ] kind: Gateway api version: gateway.networking.k8s.io/v1 [ocp_resources.resource] 2026-06-03T14:51:57.946725Z [info ] Creating gateway openshift-ingress/maas-default-gateway [conftest] name=conftest 2026-06-03T14:51:57.949832Z [info ] Create Gateway maas-default-gateway [ocp_resources Gateway] 2026-06-03T14:51:57.950143Z [info ] Posting {'apiVersion': 'gateway.networking.k8s.io/v1', 'kind': 'Gateway', 'metadata': {'name': 'maas-default-gateway', 'labels': {'app.kubernetes.io/name': 'maas', 'app.kubernetes.io/instance': 'maas-default-gateway', 'app.kubernetes.io/component': 'gateway', 'opendatahub.io/managed': 'false'}, 'annotations': {'opendatahub.io/managed': 'false', 'security.opendatahub.io/authorino-tls-bootstrap': 'true'}, 'namespace': 'openshift-ingress'}, 'spec': {'gatewayClassName': 'openshift-default', 'listeners': [{'name': 'http', 'hostname': 'maas.apps.9a96c5ff-548b-4b93-9f8e-a6229be599cf.prod.konfluxeaas.com', 'port': 80, 'protocol': 'HTTP', 'allowedRoutes': {'namespaces': {'from': 'All'}}}, {'name': 'https', 'hostname': 'maas.apps.9a96c5ff-548b-4b93-9f8e-a6229be599cf.prod.konfluxeaas.com', 'port': 443, 'protocol': 'HTTPS', 'allowedRoutes': {'namespaces': {'from': 'All'}}, 'tls': {'mode': 'Terminate', 'certificateRefs': [{'group': '', 'kind': 'Secret', 'name': 'data-science-gateway-service-tls'}]}}]}} [ocp_resources Gateway] 2026-06-03T14:51:57.960555Z [info ] Wait until Gateway maas-default-gateway is created [ocp_resources Gateway] 2026-06-03T14:51:57.960950Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-06-03T14:51:57.965531Z [info ] Elapsed time: 4.00543212890625e-05 [0:00:00.000040] [timeout_sampler] 2026-06-03T14:51:57.965731Z [info ] Executing session fixture: pytestconfig [conftest] 2026-06-03T14:51:57.965889Z [info ] Executing session fixture: use_unprivileged_client [conftest] 2026-06-03T14:51:57.966031Z [info ] Executing session fixture: kubconfig_filepath [conftest] 2026-06-03T14:51:57.966154Z [info ] Executing session fixture: cluster_authentication [conftest] 2026-06-03T14:51:57.966275Z [info ] kind: Authentication api version: config.openshift.io/v1 [ocp_resources.resource] 2026-06-03T14:51:57.969571Z [info ] Executing session fixture: is_byoidc [conftest] 2026-06-03T14:51:57.973317Z [info ] Executing session fixture: non_admin_user_password [conftest] ERROR TEST: TestMaaSApiComponentHealth.test_maas_management_state [setup] STATUS: ERROR 2026-06-03T14:51:57.997125Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (404) Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1110, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 728, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 915, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/conftest.py", line 443, in non_admin_user_password if users_Secret := list( ~~~~^ Secret.get( ^^^^^^^^^^^ ...<3 lines>... ) ^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 1690, in _get _resources = cls._prepare_resources(*args, client=_client, singular_name=singular_name, **kwargs) # type: ignore[misc] File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 881, in _prepare_resources return client.resources.get( ~~~~~~~~~~~~~~~~~~~~~ ...<2 lines>... **get_kwargs, ~~~~~~~~~~~~~ ).get(*args, **kwargs, timeout_seconds=cls.timeout_seconds) ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 112, in get return self.request('get', path, **kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 57, in inner raise api_exception(e) kubernetes.dynamic.exceptions.NotFoundError: 404 Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' Original traceback: File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) [conftest] 2026-06-03T14:51:57.997316Z [info ] Must-gather collection is enabled for /home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_management_state. [conftest] 2026-06-03T14:51:57.997471Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T14:51:57.997632Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T14:51:58.003050Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-03T14:51:58.005130Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-03T14:51:58.005547Z [info ] [generated in 0.00015s] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_management_state', 1, 0) [sqlalchemy.engine.Engine] 2026-06-03T14:51:58.005818Z [info ] [generated in 0.00015s] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_management_state', 1, 0) [sqlalchemy.engine.Engine] 2026-06-03T14:51:58.005953Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-03T14:51:58.006090Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-03T14:51:58.006224Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-03T14:51:58.006366Z [info ] kind: Subscription api version: operators.coreos.com/v1alpha1 [ocp_resources.resource] 2026-06-03T14:51:58.037732Z [info ] Checking subscription rhods-operator [utilities.infra] name=utilities.infra 2026-06-03T14:51:58.041885Z [info ] Expected CSV: rhods-operator.3.4.0 [utilities.infra] name=utilities.infra 2026-06-03T14:51:58.047090Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-03T14:51:58.052656Z [info ] kind: ClusterServiceVersion api version: operators.coreos.com/v1alpha1 [ocp_resources.resource] 2026-06-03T14:51:58.141057Z [info ] Running oc adm must-gather --dest-dir=/home/odh/opendatahub-tests/results/must-gather-collected/model_serving/maas_billing/component_health/test_maas_api_health_check/TestMaaSApiComponentHealth/test_maas_management_state/pytest_exception_interact --since=300s --image=registry.redhat.io/rhoai/odh-must-gather-rhel9@sha256:3de5ff9eee7e9b8e6650f61dbae642db3d3ccb813afb804397dc60d18f150ea6 command [pyhelper_utils.shell] 2026-06-03T14:59:15.312895Z [info ] /home/odh/opendatahub-tests/results/must-gather-collected/model_serving/maas_billing/component_health/test_maas_api_health_check/TestMaaSApiComponentHealth/test_maas_management_state/pytest_exception_interact/mg-1780498317.zip is collected successfully [utilities.must_gather_collector] name=utilities.must_gather_collector ___ ERROR at setup of TestMaaSApiComponentHealth.test_maas_management_state ____ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (404) Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1110, in execute result: FixtureValue = ihook.pytest_fixture_setup( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ fixturedef=self, request=request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pytest_asyncio/plugin.py", line 728, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 915, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/conftest.py", line 443, in non_admin_user_password if users_Secret := list( ~~~~^ Secret.get( ^^^^^^^^^^^ ...<3 lines>... ) ^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 1690, in _get _resources = cls._prepare_resources(*args, client=_client, singular_name=singular_name, **kwargs) # type: ignore[misc] File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 881, in _prepare_resources return client.resources.get( ~~~~~~~~~~~~~~~~~~~~~ ...<2 lines>... **get_kwargs, ~~~~~~~~~~~~~ ).get(*args, **kwargs, timeout_seconds=cls.timeout_seconds) ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 112, in get return self.request('get', path, **kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 57, in inner raise api_exception(e) kubernetes.dynamic.exceptions.NotFoundError: 404 Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' Original traceback: File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) _ 1 of 11 completed, 0 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 1 Error, 0 ReRun _ tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::TestMaaSApiComponentHealth::test_maas_condition_in_dsc ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ---------------------------------------------- test_maas_condition_in_dsc ---------------------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-03T14:59:15.328741Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.328937Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.329355Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.329508Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.329814Z [info ] [cached since 438.2s ago] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_condition_in_dsc', 1780498755) [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.329993Z [info ] [cached since 438.2s ago] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_condition_in_dsc', 1780498755) [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.333825Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.334062Z [info ] COMMIT  [sqlalchemy.engine.Engine] ERROR TEST: TestMaaSApiComponentHealth.test_maas_condition_in_dsc [setup] STATUS: ERROR 2026-06-03T14:59:15.350608Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (404) Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1091, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 915, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/conftest.py", line 443, in non_admin_user_password if users_Secret := list( ~~~~^ Secret.get( ^^^^^^^^^^^ ...<3 lines>... ) ^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 1690, in _get _resources = cls._prepare_resources(*args, client=_client, singular_name=singular_name, **kwargs) # type: ignore[misc] File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 881, in _prepare_resources return client.resources.get( ~~~~~~~~~~~~~~~~~~~~~ ...<2 lines>... **get_kwargs, ~~~~~~~~~~~~~ ).get(*args, **kwargs, timeout_seconds=cls.timeout_seconds) ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 112, in get return self.request('get', path, **kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 57, in inner raise api_exception(e) kubernetes.dynamic.exceptions.NotFoundError: 404 Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' Original traceback: File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) [conftest] 2026-06-03T14:59:15.351587Z [info ] Must-gather collection is enabled for /home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_condition_in_dsc. [conftest] 2026-06-03T14:59:15.351803Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.352012Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.352181Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.357736Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.357952Z [info ] [cached since 437.4s ago] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_condition_in_dsc', 1, 0) [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.358137Z [info ] [cached since 437.4s ago] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_condition_in_dsc', 1, 0) [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.358288Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.358436Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-03T14:59:15.358583Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-03T14:59:15.391865Z [info ] Checking subscription rhods-operator [utilities.infra] name=utilities.infra 2026-06-03T14:59:15.396642Z [info ] Expected CSV: rhods-operator.3.4.0 [utilities.infra] name=utilities.infra 2026-06-03T14:59:15.404723Z [info ] kind: ClusterServiceVersion api version: operators.coreos.com/v1alpha1 [ocp_resources.resource] 2026-06-03T14:59:15.485920Z [info ] Running oc adm must-gather --dest-dir=/home/odh/opendatahub-tests/results/must-gather-collected/model_serving/maas_billing/component_health/test_maas_api_health_check/TestMaaSApiComponentHealth/test_maas_condition_in_dsc/pytest_exception_interact --since=300s --image=registry.redhat.io/rhoai/odh-must-gather-rhel9@sha256:3de5ff9eee7e9b8e6650f61dbae642db3d3ccb813afb804397dc60d18f150ea6 command [pyhelper_utils.shell] 2026-06-03T15:05:28.359765Z [info ] /home/odh/opendatahub-tests/results/must-gather-collected/model_serving/maas_billing/component_health/test_maas_api_health_check/TestMaaSApiComponentHealth/test_maas_condition_in_dsc/pytest_exception_interact/mg-1780498755.zip is collected successfully [utilities.must_gather_collector] name=utilities.must_gather_collector ___ ERROR at setup of TestMaaSApiComponentHealth.test_maas_condition_in_dsc ____ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (404) Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1091, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 915, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/conftest.py", line 443, in non_admin_user_password if users_Secret := list( ~~~~^ Secret.get( ^^^^^^^^^^^ ...<3 lines>... ) ^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 1690, in _get _resources = cls._prepare_resources(*args, client=_client, singular_name=singular_name, **kwargs) # type: ignore[misc] File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 881, in _prepare_resources return client.resources.get( ~~~~~~~~~~~~~~~~~~~~~ ...<2 lines>... **get_kwargs, ~~~~~~~~~~~~~ ).get(*args, **kwargs, timeout_seconds=cls.timeout_seconds) ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 112, in get return self.request('get', path, **kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 57, in inner raise api_exception(e) kubernetes.dynamic.exceptions.NotFoundError: 404 Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' Original traceback: File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) _ 2 of 11 completed, 0 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 2 Error, 0 ReRun _ tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::TestMaaSApiComponentHealth::test_maas_api_deployment_available ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ------------------------------------------ test_maas_api_deployment_available ------------------------------------------ -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-03T15:05:28.364463Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.364715Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.364921Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.365114Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.365338Z [info ] [cached since 811.2s ago] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_api_deployment_available', 1780499128) [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.365554Z [info ] [cached since 811.2s ago] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_api_deployment_available', 1780499128) [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.365784Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.366324Z [info ] COMMIT  [sqlalchemy.engine.Engine] ERROR TEST: TestMaaSApiComponentHealth.test_maas_api_deployment_available [setup] STATUS: ERROR 2026-06-03T15:05:28.375037Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (404) Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1091, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 915, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/conftest.py", line 443, in non_admin_user_password if users_Secret := list( ~~~~^ Secret.get( ^^^^^^^^^^^ ...<3 lines>... ) ^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 1690, in _get _resources = cls._prepare_resources(*args, client=_client, singular_name=singular_name, **kwargs) # type: ignore[misc] File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 881, in _prepare_resources return client.resources.get( ~~~~~~~~~~~~~~~~~~~~~ ...<2 lines>... **get_kwargs, ~~~~~~~~~~~~~ ).get(*args, **kwargs, timeout_seconds=cls.timeout_seconds) ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 112, in get return self.request('get', path, **kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 57, in inner raise api_exception(e) kubernetes.dynamic.exceptions.NotFoundError: 404 Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' Original traceback: File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) [conftest] 2026-06-03T15:05:28.375475Z [info ] Must-gather collection is enabled for /home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_api_deployment_available. [conftest] 2026-06-03T15:05:28.375703Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.375897Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.379365Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.379581Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.380073Z [info ] [cached since 810.4s ago] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_api_deployment_available', 1, 0) [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.380317Z [info ] [cached since 810.4s ago] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_api_deployment_available', 1, 0) [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.380548Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.381005Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-03T15:05:28.389008Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-03T15:05:28.423373Z [info ] Checking subscription rhods-operator [utilities.infra] name=utilities.infra 2026-06-03T15:05:28.428314Z [info ] Expected CSV: rhods-operator.3.4.0 [utilities.infra] name=utilities.infra 2026-06-03T15:05:28.436387Z [info ] kind: ClusterServiceVersion api version: operators.coreos.com/v1alpha1 [ocp_resources.resource] 2026-06-03T15:05:28.500932Z [info ] Running oc adm must-gather --dest-dir=/home/odh/opendatahub-tests/results/must-gather-collected/model_serving/maas_billing/component_health/test_maas_api_health_check/TestMaaSApiComponentHealth/test_maas_api_deployment_available/pytest_exception_interact --since=300s --image=registry.redhat.io/rhoai/odh-must-gather-rhel9@sha256:3de5ff9eee7e9b8e6650f61dbae642db3d3ccb813afb804397dc60d18f150ea6 command [pyhelper_utils.shell] 2026-06-03T15:11:38.746726Z [info ] /home/odh/opendatahub-tests/results/must-gather-collected/model_serving/maas_billing/component_health/test_maas_api_health_check/TestMaaSApiComponentHealth/test_maas_api_deployment_available/pytest_exception_interact/mg-1780499128.zip is collected successfully [utilities.must_gather_collector] name=utilities.must_gather_collector _ ERROR at setup of TestMaaSApiComponentHealth.test_maas_api_deployment_available _ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (404) Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1091, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 915, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/conftest.py", line 443, in non_admin_user_password if users_Secret := list( ~~~~^ Secret.get( ^^^^^^^^^^^ ...<3 lines>... ) ^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 1690, in _get _resources = cls._prepare_resources(*args, client=_client, singular_name=singular_name, **kwargs) # type: ignore[misc] File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 881, in _prepare_resources return client.resources.get( ~~~~~~~~~~~~~~~~~~~~~ ...<2 lines>... **get_kwargs, ~~~~~~~~~~~~~ ).get(*args, **kwargs, timeout_seconds=cls.timeout_seconds) ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 112, in get return self.request('get', path, **kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 57, in inner raise api_exception(e) kubernetes.dynamic.exceptions.NotFoundError: 404 Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' Original traceback: File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) _ 3 of 11 completed, 0 Pass, 0 Fail, 0 Skip, 0 XPass, 0 XFail, 3 Error, 0 ReRun _ tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::TestMaaSApiComponentHealth::test_maas_api_pods_health ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ---------------------------------------------- test_maas_api_pods_health ---------------------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-06-03T15:11:38.753927Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.754143Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.754319Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.754473Z [info ] INSERT INTO "OpenDataHubTestTable" (test_name, start_time) VALUES (?, ?) [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.754892Z [info ] [cached since 1182s ago] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_api_pods_health', 1780499498) [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.755068Z [info ] [cached since 1182s ago] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_api_pods_health', 1780499498) [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.756764Z [info ] COMMIT  [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.756993Z [info ] COMMIT  [sqlalchemy.engine.Engine] ERROR TEST: TestMaaSApiComponentHealth.test_maas_api_pods_health [setup] STATUS: ERROR 2026-06-03T15:11:38.767209Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (404) Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 353, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 245, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup item.session._setupstate.setup(item) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 523, in setup col.setup() ~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1723, in setup self._request._fillfixtures() ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures item.funcargs[argname] = self.getfixturevalue(argname) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue fixturedef = self._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1068, in execute fixturedef = request._get_active_fixturedef(argname) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef fixturedef.execute(request=subrequest) ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1091, in execute raise exc.with_traceback(exc_tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/fixtures.py", line 915, in call_fixture_func fixture_result = fixturefunc(**kwargs) File "/home/odh/opendatahub-tests/tests/conftest.py", line 443, in non_admin_user_password if users_Secret := list( ~~~~^ Secret.get( ^^^^^^^^^^^ ...<3 lines>... ) ^ ): ^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 1690, in _get _resources = cls._prepare_resources(*args, client=_client, singular_name=singular_name, **kwargs) # type: ignore[misc] File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 881, in _prepare_resources return client.resources.get( ~~~~~~~~~~~~~~~~~~~~~ ...<2 lines>... **get_kwargs, ~~~~~~~~~~~~~ ).get(*args, **kwargs, timeout_seconds=cls.timeout_seconds) ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 112, in get return self.request('get', path, **kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 57, in inner raise api_exception(e) kubernetes.dynamic.exceptions.NotFoundError: 404 Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '5cc877a2-a6e0-4a95-8f31-ecd34e4683e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'd3d13330-56cd-4207-b96f-468c303c42bf', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4ef1427-b911-4981-859e-f3d2317e93b3', 'Date': 'Wed, 03 Jun 2026 14:51:57 GMT', 'Content-Length': '190'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"openldap\\" not found","reason":"NotFound","details":{"name":"openldap","kind":"secrets"},"code":404}\n' Original traceback: File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 55, in inner resp = func(self, *args, **kwargs) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/dynamic/client.py", line 277, in request api_response = self.client.call_api( path, ...<11 lines>... _request_timeout=params.get('_request_timeout') ) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ path_params, query_params, header_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<2 lines>... _return_http_data_only, collection_formats, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content, _request_timeout, _host) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( method, url, query_params=query_params, headers=header_params, post_params=post_params, body=body, _preload_content=_preload_content, _request_timeout=_request_timeout) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, ~~~~~~~~~~~~~~~~~~~~^^^^^ query_params=query_params, ^^^^^^^^^^^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ headers=headers) ^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 244, in GET return self.request("GET", url, ~~~~~~~~~~~~^^^^^^^^^^^^ headers=headers, ^^^^^^^^^^^^^^^^ _preload_content=_preload_content, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _request_timeout=_request_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ query_params=query_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/kubernetes/client/rest.py", line 238, in request raise ApiException(http_resp=r) [conftest] 2026-06-03T15:11:38.771703Z [info ] Must-gather collection is enabled for /home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_api_pods_health. [conftest] 2026-06-03T15:11:38.771857Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.772003Z [info ] BEGIN (implicit)  [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.772147Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.772276Z [info ] SELECT "OpenDataHubTestTable".start_time AS "OpenDataHubTestTable_start_time" FROM "OpenDataHubTestTable" WHERE "OpenDataHubTestTable".test_name = ? LIMIT ? OFFSET ? [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.772436Z [info ] [cached since 1181s ago] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_api_pods_health', 1, 0) [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.772585Z [info ] [cached since 1181s ago] ('/home/odh/opendatahub-tests/tests/model_serving/maas_billing/component_health/test_maas_api_health_check.py::test_maas_api_pods_health', 1, 0) [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.772732Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.772851Z [info ] ROLLBACK  [sqlalchemy.engine.Engine] 2026-06-03T15:11:38.772973Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-06-03T15:11:38.799713Z [info ] Checking subscription rhods-operator [utilities.infra] name=utilities.infra 2026-06-03T15:11:38.804250Z [info ] Expected CSV: rhods-operator.3.4.0 [utilities.infra] name=utilities.infra 2026-06-03T15:11:38.813061Z [info ] kind: ClusterServiceVersion api version: operators.coreos.com/v1alpha1 [ocp_resources.resource] 2026-06-03T15:11:38.895157Z [info ] Running oc adm must-gather --dest-dir=/home/odh/opendatahub-tests/results/must-gather-collected/model_serving/maas_billing/component_health/test_maas_api_health_check/TestMaaSApiComponentHealth/test_maas_api_pods_health/pytest_exception_interact --since=300s --image=registry.redhat.io/rhoai/odh-must-gather-rhel9@sha256:3de5ff9eee7e9b8e6650f61dbae642db3d3ccb813afb804397dc60d18f150ea6 command [pyhelper_utils.shell] /usr/lib64/python3.14/multiprocessing/resource_tracker.py:475: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown: {'/mp-83pgoyu_', '/mp-e9dvodbm', '/mp-80z1gawb'} warnings.warn(