--- pytest attempt 2026-09-10T06:52:00.123172+00:00 (/.local/bin/uv --directory /home/odh/opendatahub-tests...) --- 2026-09-10T06:52:02.334913Z [info ] Writing tests log to /home/odh/opendatahub-tests/results/pytest-tests.log [conftest] ============================= test session starts ============================== platform linux -- Python 3.14.7, pytest-9.1.1, pluggy-1.6.0 -- /home/odh/opendatahub-tests/.venv/bin/python3 cachedir: .pytest_cache metadata: {'Python': '3.14.7', 'Platform': 'Linux-5.14.0-570.132.1.el9_6.x86_64-x86_64-with-glibc2.42', 'Packages': {'pytest': '9.1.1', 'pluggy': '1.6.0'}, 'Plugins': {'anyio': '4.15.1', 'langsmith': '0.12.2', 'asyncio': '1.4.0', 'dependency': '0.6.1', 'html': '4.2.0', 'metadata': '3.1.1', 'order': '1.5.0', 'progress': '1.4.0', 'testconfig': '0.2.0', 'xdist': '3.8.0', 'syrupy': '6.0.0'}} rootdir: /home/odh/opendatahub-tests configfile: pytest.ini plugins: anyio-4.15.1, langsmith-0.12.2, asyncio-1.4.0, dependency-0.6.1, html-4.2.0, metadata-3.1.1, order-1.5.0, progress-1.4.0, testconfig-0.2.0, xdist-3.8.0, syrupy-6.0.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collecting ... 2026-09-10T06:52:03.191933Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-09-10T06:52:04.602166Z [info ] kind: ClusterPolicy api version: kyverno.io/v2beta1 [ocp_resources.resource] 2026-09-10T06:52:04.631478Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-09-10T06:52:04.699080Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-09-10T06:52:04.771431Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] collected 398 items / 377 deselected / 21 selected tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_e2e.py::TestE2eLifecycle::test_e2e_001_successful_evaluation_lifecycle  ------------------------------------- test_e2e_001_successful_evaluation_lifecycle ------------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- 2026-09-10T06:52:04.802760Z [info ] Executing session fixture: event_loop_policy [conftest] 2026-09-10T06:52:04.802928Z [info ] Executing session fixture: _syrupy_apply_ide_patches [conftest] 2026-09-10T06:52:04.803067Z [info ] Executing session fixture: admin_client [conftest] 2026-09-10T06:52:04.803212Z [info ] Trying to get client via new_client_from_config [ocp_resources.resource] 2026-09-10T06:52:04.809438Z [info ] Executing session fixture: tmp_path_factory [conftest] 2026-09-10T06:52:04.810017Z [info ] Executing session fixture: tests_tmp_dir [conftest] 2026-09-10T06:52:04.810704Z [info ] Executing session fixture: os_path_environment [conftest] 2026-09-10T06:52:04.810836Z [info ] Executing session fixture: tmpdir_factory [conftest] 2026-09-10T06:52:04.810967Z [info ] Executing session fixture: bin_directory [conftest] 2026-09-10T06:52:04.812382Z [info ] Executing session fixture: oc_binary_path [conftest] 2026-09-10T06:52:04.811446Z [warning ] Using previously installed: /workspace/tests-shared/tests-payload/.tools/bin/oc [tests.conftest] name=tests.conftest 2026-09-10T06:52:04.812683Z [info ] Executing session fixture: bin_directory_to_os_path [conftest] 2026-09-10T06:52:04.811579Z [info ] OC binary path: /workspace/tests-shared/tests-payload/.tools/bin/oc [tests.conftest] name=tests.conftest 2026-09-10T06:52:04.811621Z [info ] Adding /tmp/pytest-CNwLJSa3c8E2UKj8CtwqYu/bin0 to $PATH [tests.conftest] name=tests.conftest 2026-09-10T06:52:04.813016Z [info ] Executing session fixture: nodes [conftest] 2026-09-10T06:52:04.903971Z [info ] Executing session fixture: record_testsuite_property [conftest] 2026-09-10T06:52:04.904137Z [info ] Executing session fixture: junitxml_plugin [conftest] 2026-09-10T06:52:04.904289Z [info ] Executing session fixture: cluster_sanity_scope_session [conftest] 2026-09-10T06:52:04.904553Z [info ] Executing session fixture: dsci_resource [conftest] 2026-09-10T06:52:04.904706Z [info ] kind: DSCInitialization api version: dscinitialization.opendatahub.io/v2 [ocp_resources.resource] 2026-09-10T06:52:04.920998Z [info ] Executing session fixture: dsc_resource [conftest] 2026-09-10T06:52:04.921160Z [info ] kind: DataScienceCluster api version: datasciencecluster.opendatahub.io/v2 [ocp_resources.resource] 2026-09-10T06:52:04.940954Z [info ] Check cluster sanity.  [utilities.infra] name=utilities.infra 2026-09-10T06:52:04.941847Z [info ] Verify all nodes are in a healthy condition. [ocp_utilities.infra] 2026-09-10T06:52:04.994913Z [info ] Verify all nodes are schedulable. [ocp_utilities.infra] 2026-09-10T06:52:05.047477Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsci_status_ready Kwargs: {'dsci_resource': }) [timeout_sampler] 2026-09-10T06:52:05.046757Z [info ] Wait for DSCI default-dsci to be in Ready status. [utilities.infra] name=utilities.infra 2026-09-10T06:52:05.047802Z [info ] Get DSCInitialization default-dsci status [ocp_resources DSCInitialization] 2026-09-10T06:52:05.064770Z [success ] Function: utilities.infra.wait_for_dsci_status_ready succeeded after 0.00011444091796875 [0:00:00.000114]. [timeout_sampler] 2026-09-10T06:52:05.064902Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: utilities.infra.wait_for_dsc_status_ready Kwargs: {'dsc_resource': }) [timeout_sampler] 2026-09-10T06:52:05.064078Z [info ] Wait for DSC default-dsc are Ready. [utilities.infra] name=utilities.infra 2026-09-10T06:52:05.065126Z [info ] Get DataScienceCluster default-dsc status [ocp_resources DataScienceCluster] 2026-09-10T06:52:05.086170Z [success ] Function: utilities.infra.wait_for_dsc_status_ready succeeded after 6.461143493652344e-05 [0:00:00.000065]. [timeout_sampler] 2026-09-10T06:52:05.086320Z [info ] Executing session fixture: autouse_fixtures [conftest] 2026-09-10T06:52:05.086485Z [info ] Executing session fixture: lifecycle_signals_cp_namespace [conftest] 2026-09-10T06:52:05.104605Z [info ] Create Namespace test-k8s-lifecycle-signals-cp [ocp_resources Namespace] 2026-09-10T06:52:05.104821Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'test-k8s-lifecycle-signals-cp'}, 'spec': {}} [ocp_resources Namespace] 2026-09-10T06:52:05.161011Z [info ] Wait for Namespace test-k8s-lifecycle-signals-cp status to be Active [ocp_resources Namespace] 2026-09-10T06:52:05.161139Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-09-10T06:52:05.183698Z [info ] Status of Namespace test-k8s-lifecycle-signals-cp is Active [ocp_resources Namespace] 2026-09-10T06:52:05.183908Z [success ] Function: lambda: self.exists succeeded after 0.00010061264038085938 [0:00:00.000101]. [timeout_sampler] 2026-09-10T06:52:05.184025Z [info ] Executing session fixture: lifecycle_signals_evalhub_cr [conftest] 2026-09-10T06:52:05.184129Z [info ] kind: CustomResourceDefinition api version: apiextensions.k8s.io/v1 [ocp_resources.resource] 2026-09-10T06:52:05.240663Z [info ] kind: EvalHub api version: trustyai.opendatahub.io/v1 [ocp_resources.resource] 2026-09-10T06:52:05.240879Z [info ] Create EvalHub evalhub-ls  [ocp_resources EvalHub] 2026-09-10T06:52:05.241002Z [info ] Posting {'apiVersion': 'trustyai.opendatahub.io/v1', 'kind': 'EvalHub', 'metadata': {'name': 'evalhub-ls', 'namespace': 'test-k8s-lifecycle-signals-cp'}, 'spec': {'database': {'type': 'sqlite'}}} [ocp_resources EvalHub] 2026-09-10T06:52:05.283261Z [info ] Wait until EvalHub evalhub-ls is created [ocp_resources EvalHub] 2026-09-10T06:52:05.283497Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-09-10T06:52:05.302549Z [success ] Function: lambda: self.exists succeeded after 7.271766662597656e-05 [0:00:00.000073]. [timeout_sampler] 2026-09-10T06:52:05.302693Z [info ] Executing session fixture: lifecycle_signals_deployment [conftest] 2026-09-10T06:52:05.302817Z [info ] kind: Deployment api version: apps/v1 [ocp_resources.resource] 2026-09-10T06:52:05.302931Z [info ] Wait for Deployment evalhub-ls to be deployed: True [ocp_resources Deployment] 2026-09-10T06:52:05.303032Z [info ] Waiting for 300 seconds [0:05:00], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-09-10T06:52:08.395299Z [success ] Function: lambda: self.exists succeeded after 3.0589828491210938 [0:00:03.058983]. [timeout_sampler] 2026-09-10T06:52:08.395506Z [info ] Waiting for 296.9073238372803 seconds [0:04:56.907324], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-09-10T06:52:19.794905Z [success ] Function: lambda: self.instance succeeded after 11.36726689338684 [0:00:11.367267]. [timeout_sampler] 2026-09-10T06:52:19.795085Z [info ] Executing session fixture: lifecycle_signals_route [conftest] 2026-09-10T06:52:19.795232Z [info ] kind: Route api version: route.openshift.io/v1 [ocp_resources.resource] 2026-09-10T06:52:19.815182Z [info ] Executing session fixture: lifecycle_signals_ca_bundle_file [conftest] 2026-09-10T06:52:19.842748Z [info ] kind: IngressController api version: operator.openshift.io/v1 [ocp_resources.resource] 2026-09-10T06:52:19.860498Z [info ] Using router certificate secret: default-ingress-cert [utilities.certificates_utils] name=utilities.certificates_utils 2026-09-10T06:52:19.877765Z [info ] Executing session fixture: lifecycle_signals_ready [conftest] 2026-09-10T06:52:19.895873Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.conftest..lambda: host.lifecycle_signals_ca_bundle_file.url.probe_evalhub_health_endpoint) [timeout_sampler] 2026-09-10T06:52:19.934222Z [info ] EvalHub at evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev is healthy [tests.ai_safety.evalhub.k8s_lifecycle_signals.conftest] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.conftest 2026-09-10T06:52:19.935812Z [success ] Function: lambda: host.lifecycle_signals_ca_bundle_file.url.probe_evalhub_health_endpoint succeeded after 0.00013399124145507812 [0:00:00.000134]. [timeout_sampler] 2026-09-10T06:52:19.935966Z [info ] Executing session fixture: lifecycle_signals_namespace [conftest] 2026-09-10T06:52:19.953811Z [info ] Create Namespace test-k8s-lifecycle-signals [ocp_resources Namespace] 2026-09-10T06:52:19.954188Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'test-k8s-lifecycle-signals', 'labels': {'evalhub.trustyai.opendatahub.io/tenant': 'true'}}, 'spec': {}} [ocp_resources Namespace] 2026-09-10T06:52:20.013062Z [info ] Wait for Namespace test-k8s-lifecycle-signals status to be Active [ocp_resources Namespace] 2026-09-10T06:52:20.013226Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_for_status.lambda: self.exists) [timeout_sampler] 2026-09-10T06:52:20.030745Z [info ] Status of Namespace test-k8s-lifecycle-signals is Active [ocp_resources Namespace] 2026-09-10T06:52:20.030908Z [success ] Function: lambda: self.exists succeeded after 4.9114227294921875e-05 [0:00:00.000049]. [timeout_sampler] 2026-09-10T06:52:20.031069Z [info ] Executing session fixture: lifecycle_signals_token [conftest] 2026-09-10T06:52:20.031213Z [info ] Create ServiceAccount evalhub-lifecycle-user [ocp_resources ServiceAccount] 2026-09-10T06:52:20.031358Z [info ] Posting {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'evalhub-lifecycle-user', 'namespace': 'test-k8s-lifecycle-signals'}} [ocp_resources ServiceAccount] 2026-09-10T06:52:20.066678Z [info ] Wait until ServiceAccount evalhub-lifecycle-user is created [ocp_resources ServiceAccount] 2026-09-10T06:52:20.066852Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-09-10T06:52:20.084794Z [success ] Function: lambda: self.exists succeeded after 4.482269287109375e-05 [0:00:00.000045]. [timeout_sampler] 2026-09-10T06:52:20.084982Z [info ] kind: Role api version: rbac.authorization.k8s.io/v1 [ocp_resources.resource] 2026-09-10T06:52:20.085132Z [info ] Create Role evalhub-lifecycle-user-role [ocp_resources Role] 2026-09-10T06:52:20.085277Z [info ] Posting {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'Role', 'metadata': {'name': 'evalhub-lifecycle-user-role', 'namespace': 'test-k8s-lifecycle-signals'}, 'rules': [{'apiGroups': ['trustyai.opendatahub.io'], 'resources': ['evaluations', 'collections', 'providers'], 'verbs': ['get', 'list', 'create', 'update', 'delete']}, {'apiGroups': ['mlflow.kubeflow.org'], 'resources': ['experiments'], 'verbs': ['create', 'get']}]} [ocp_resources Role] 2026-09-10T06:52:20.119649Z [info ] Wait until Role evalhub-lifecycle-user-role is created [ocp_resources Role] 2026-09-10T06:52:20.119817Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-09-10T06:52:20.136842Z [success ] Function: lambda: self.exists succeeded after 5.316734313964844e-05 [0:00:00.000053]. [timeout_sampler] 2026-09-10T06:52:20.137002Z [info ] kind: RoleBinding api version: rbac.authorization.k8s.io/v1 [ocp_resources.resource] 2026-09-10T06:52:20.137144Z [info ] Create RoleBinding evalhub-lifecycle-user-binding [ocp_resources RoleBinding] 2026-09-10T06:52:20.137286Z [info ] Posting {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'RoleBinding', 'metadata': {'name': 'evalhub-lifecycle-user-binding', 'namespace': 'test-k8s-lifecycle-signals'}, 'subjects': [{'kind': 'ServiceAccount', 'name': 'evalhub-lifecycle-user', 'namespace': 'test-k8s-lifecycle-signals'}], 'roleRef': {'kind': 'Role', 'name': 'evalhub-lifecycle-user-role', 'apiGroup': 'rbac.authorization.k8s.io'}} [ocp_resources RoleBinding] 2026-09-10T06:52:20.271420Z [info ] Wait until RoleBinding evalhub-lifecycle-user-binding is created [ocp_resources RoleBinding] 2026-09-10T06:52:20.271567Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-09-10T06:52:20.288530Z [success ] Function: lambda: self.exists succeeded after 5.91278076171875e-05 [0:00:00.000059]. [timeout_sampler] 2026-09-10T06:52:20.288681Z [info ] Create RoleBinding evalhub-lifecycle-jobs-writer-binding [ocp_resources RoleBinding] 2026-09-10T06:52:20.288827Z [info ] Posting {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'RoleBinding', 'metadata': {'name': 'evalhub-lifecycle-jobs-writer-binding', 'namespace': 'test-k8s-lifecycle-signals'}, 'subjects': [{'kind': 'ServiceAccount', 'name': 'evalhub-lifecycle-user', 'namespace': 'test-k8s-lifecycle-signals'}], 'roleRef': {'kind': 'ClusterRole', 'name': 'trustyai-service-operator-evalhub-jobs-writer', 'apiGroup': 'rbac.authorization.k8s.io'}} [ocp_resources RoleBinding] 2026-09-10T06:52:20.370320Z [info ] Wait until RoleBinding evalhub-lifecycle-jobs-writer-binding is created [ocp_resources RoleBinding] 2026-09-10T06:52:20.370492Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait.lambda: self.exists) [timeout_sampler] 2026-09-10T06:52:20.386959Z [success ] Function: lambda: self.exists succeeded after 4.6253204345703125e-05 [0:00:00.000046]. [timeout_sampler] 2026-09-10T06:52:20.410138Z [info ] Executing session fixture: lifecycle_signals_tenant_rbac [conftest] 2026-09-10T06:52:20.410311Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: tests.ai_safety.evalhub.utils.tenant_rbac_ready Kwargs: {'admin_client': , 'namespace': 'test-k8s-lifecycle-signals', 'evalhub_instance_name': 'evalhub-ls'}) [timeout_sampler] 2026-09-10T06:52:26.230513Z [info ] Operator RBAC provisioned in test-k8s-lifecycle-signals [tests.ai_safety.evalhub.k8s_lifecycle_signals.conftest] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.conftest 2026-09-10T06:52:26.233264Z [success ] Function: tests.ai_safety.evalhub.utils.tenant_rbac_ready succeeded after 5.286808013916016 [0:00:05.286808]. [timeout_sampler] 2026-09-10T06:52:26.233442Z [info ] Executing session fixture: lifecycle_signals_vllm_deployment [conftest] 2026-09-10T06:52:26.233587Z [info ] kind: Deployment api version: apps/v1 --- [DuplicateFilter: Last log `kind: RoleBinding api version: rbac.authorization.k8s.io/v1` repeated 2 times] [ocp_resources.resource] 2026-09-10T06:52:26.233700Z [info ] Create Deployment vllm-emulator [ocp_resources Deployment] 2026-09-10T06:52:26.233816Z [info ] Posting {'apiVersion': 'apps/v1', 'kind': 'Deployment', 'metadata': {'name': 'vllm-emulator', 'labels': {'app': 'vllm-emulator'}, 'namespace': 'test-k8s-lifecycle-signals'}, 'spec': {'selector': {'matchLabels': {'app': 'vllm-emulator'}}, 'template': {'metadata': {'labels': {'app': 'vllm-emulator'}, 'name': 'vllm-emulator'}, 'spec': {'containers': [{'image': 'quay.io/trustyai_testing/vllm_emulator@sha256:32b5f26b5ec1c5c8052afa26c6d9769dbc864df27a058716c8df62d827cf1d07', 'name': 'vllm-emulator', 'ports': [{'containerPort': 8000, 'protocol': 'TCP'}], 'readinessProbe': {'tcpSocket': {'port': 8000}, 'initialDelaySeconds': 5, 'periodSeconds': 5, 'timeoutSeconds': 3, 'failureThreshold': 6}, 'securityContext': {'allowPrivilegeEscalation': False, 'runAsNonRoot': True, 'capabilities': {'drop': ['ALL']}, 'seccompProfile': {'type': 'RuntimeDefault'}}}]}}, 'replicas': 1}} [ocp_resources Deployment] 2026-09-10T06:52:26.279146Z [info ] Wait for Deployment vllm-emulator to be deployed: True [ocp_resources Deployment] 2026-09-10T06:52:26.279295Z [info ] Waiting for 300 seconds [0:05:00], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.exists) [timeout_sampler] 2026-09-10T06:52:26.295976Z [success ] Function: lambda: self.exists succeeded after 4.57763671875e-05 [0:00:00.000046]. [timeout_sampler] 2026-09-10T06:52:26.300664Z [info ] Waiting for 299.98340106010437 seconds [0:04:59.983401], retry every 1 seconds. (Function: ocp_resources.deployment.wait_for_replicas.lambda: self.instance) [timeout_sampler] 2026-09-10T06:52:42.605986Z [success ] Function: lambda: self.instance succeeded after 16.29114317893982 [0:00:16.291143]. [timeout_sampler] 2026-09-10T06:52:42.606218Z [info ] Executing session fixture: lifecycle_signals_vllm_service [conftest] 2026-09-10T06:52:42.606336Z [info ] Create Service vllm-emulator-service [ocp_resources Service] 2026-09-10T06:52:42.606488Z [info ] Posting {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'vllm-emulator-service', 'namespace': 'test-k8s-lifecycle-signals'}, 'spec': {'ports': [{'name': 'vllm-emulator-endpoint', 'port': 8000, 'protocol': 'TCP', 'targetPort': 8000}], 'selector': {'app': 'vllm-emulator'}}} [ocp_resources Service] --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-10T06:52:42.667602Z [info ] Submitting evaluation job to https://evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev/api/v1/evaluations/jobs for tenant test-k8s-lifecycle-signals [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:52:42.847335Z [info ] Job submitted: 0eee3b6d-4b7e-4d6f-bc84-9184b2cd2c8d [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:52:42.848635Z [info ] Waiting for 600 seconds [0:10:00], retry every 3 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find) [timeout_sampler] 2026-09-10T06:52:42.848763Z [info ] kind: Job api version: batch/v1 [ocp_resources.resource] 2026-09-10T06:52:45.884081Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find succeeded after 3.017247200012207 [0:00:03.017247]. [timeout_sampler] 2026-09-10T06:52:45.884327Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label) [timeout_sampler] 2026-09-10T06:54:46.746545Z [error ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label timed out after 120 [0:02:00]. [timeout_sampler] FAILED ________ TestE2eLifecycle.test_e2e_001_successful_evaluation_lifecycle _________ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_e2e.py", line 98, in test_e2e_001_successful_evaluation_lifecycle wait_for_job_label( ~~~~~~~~~~~~~~~~~~^ admin_client=admin_client, ^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... timeout=LIFECYCLE_JOB_LABEL_TIMEOUT, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/k8s_lifecycle_signals/utils.py", line 126, in wait_for_job_label for value in TimeoutSampler(wait_timeout=timeout, sleep=sleep, func=_get_label): ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 346, in __iter__ raise TimeoutExpiredError( ...<3 lines>... ) timeout_sampler.TimeoutExpiredError: Timed Out: 120 Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label Last exception: N/A. Elapsed time: 120 [0:02:00] _ 1 of 21 completed, 0 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_e2e.py::TestE2eLifecycle::test_e2e_002_server_reported_failure_lifecycle  TEST: TestE2eLifecycle.test_e2e_001_successful_evaluation_lifecycle STATUS: FAILED 2026-09-10T06:54:46.756262Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_e2e.py", line 98, in test_e2e_001_successful_evaluation_lifecycle wait_for_job_label( ~~~~~~~~~~~~~~~~~~^ admin_client=admin_client, ^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... timeout=LIFECYCLE_JOB_LABEL_TIMEOUT, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/k8s_lifecycle_signals/utils.py", line 126, in wait_for_job_label for value in TimeoutSampler(wait_timeout=timeout, sleep=sleep, func=_get_label): ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 346, in __iter__ raise TimeoutExpiredError( ...<3 lines>... ) timeout_sampler.TimeoutExpiredError: Timed Out: 120 Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label Last exception: N/A. Elapsed time: 120 [0:02:00] [conftest] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ------------------------------------ test_e2e_002_server_reported_failure_lifecycle ------------------------------------ -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-10T06:54:46.777306Z [info ] Submitting evaluation job to https://evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev/api/v1/evaluations/jobs for tenant test-k8s-lifecycle-signals [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:54:46.913662Z [info ] Job submitted: 317ce57e-102e-4f89-a511-98277b010a0b [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:54:46.915171Z [info ] Waiting for 600 seconds [0:10:00], retry every 3 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find) [timeout_sampler] 2026-09-10T06:54:49.951930Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find succeeded after 3.0172860622406006 [0:00:03.017286]. [timeout_sampler] 2026-09-10T06:54:49.950621Z [info ] Waiting for job 317ce57e-102e-4f89-a511-98277b010a0b to complete (timeout=600s) [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:54:49.952564Z [info ] Waiting for 600 seconds [0:10:00], retry every 10 seconds. (Function: tests.ai_safety.evalhub.utils.get_job_status Kwargs: {'host': 'evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev', 'token': '***', 'ca_bundle_file': '/tmp/pytest-CNwLJSa3c8E2UKj8CtwqYu/tests0/openshift_ca.crt', 'tenant': 'test-k8s-lifecycle-signals', 'job_id': '317ce57e-102e-4f89-a511-98277b010a0b'}) [timeout_sampler] 2026-09-10T06:54:49.979946Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:55:00.049372Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:55:10.120510Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:55:20.153558Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:55:30.177286Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:55:40.248812Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:55:50.281892Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:56:00.298070Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:56:10.329197Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:56:20.354773Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:56:30.385817Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:56:40.411509Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:56:50.457071Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:00.572112Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:10.594194Z [info ] Job 317ce57e-102e-4f89-a511-98277b010a0b state: failed [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:10.595293Z [success ] Function: tests.ai_safety.evalhub.utils.get_job_status succeeded after 140.62162613868713 [0:02:20.621626]. [timeout_sampler] 2026-09-10T06:57:10.595483Z [info ] Listing events  [ocp_resources.event] 2026-09-10T06:57:10.667559Z [info ] Waiting for 240 seconds [0:04:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event) [timeout_sampler] 2026-09-10T06:57:10.682710Z [info ] Event EvaluationFailed emitted for job 317ce57e-102e-4f89-a511-98-46a0fd43-6814-4922-935b-f9b0a1aab454 (elapsed=0.0s) [tests.ai_safety.evalhub.k8s_lifecycle_signals.utils] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.utils 2026-09-10T06:57:10.684130Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event succeeded after 0.00010395050048828125 [0:00:00.000104]. [timeout_sampler] 2026-09-10T06:57:10.684252Z [info ] Waiting for 240 seconds [0:04:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event) [timeout_sampler] 2026-09-10T06:57:10.701021Z [info ] Event EvaluationFailed emitted for job 317ce57e-102e-4f89-a511-98-46a0fd43-6814-4922-935b-f9b0a1aab454 (elapsed=0.0s) [tests.ai_safety.evalhub.k8s_lifecycle_signals.utils] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.utils 2026-09-10T06:57:10.702120Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event succeeded after 3.5762786865234375e-05 [0:00:00.000036]. [timeout_sampler] 2026-09-10T06:57:10.702239Z [info ] Waiting for 120 seconds [0:02:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label) [timeout_sampler] 2026-09-10T06:57:10.733950Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label succeeded after 2.7894973754882812e-05 [0:00:00.000028]. [timeout_sampler] PASSED _ 2 of 21 completed, 1 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_e2e.py::TestE2eLifecycle::test_e2e_003_infrastructure_failure_lifecycle  TEST: TestE2eLifecycle.test_e2e_002_server_reported_failure_lifecycle STATUS: PASSED ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ------------------------------------ test_e2e_003_infrastructure_failure_lifecycle ------------------------------------ -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-10T06:57:10.801738Z [info ] Submitting evaluation job to https://evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev/api/v1/evaluations/jobs for tenant test-k8s-lifecycle-signals [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:10.829117Z [info ] Job submitted: 3b22e4b8-2ba0-417a-95a0-e23b8584daaf [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:10.830189Z [info ] Waiting for 600 seconds [0:10:00], retry every 3 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find) [timeout_sampler] 2026-09-10T06:57:13.866249Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find succeeded after 3.0187437534332275 [0:00:03.018744]. [timeout_sampler] 2026-09-10T06:57:13.865414Z [info ] Waiting for job 3b22e4b8-2ba0-417a-95a0-e23b8584daaf to complete (timeout=600s) [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:13.866606Z [info ] Waiting for 600 seconds [0:10:00], retry every 10 seconds. (Function: tests.ai_safety.evalhub.utils.get_job_status Kwargs: {'host': 'evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev', 'token': '***', 'ca_bundle_file': '/tmp/pytest-CNwLJSa3c8E2UKj8CtwqYu/tests0/openshift_ca.crt', 'tenant': 'test-k8s-lifecycle-signals', 'job_id': '3b22e4b8-2ba0-417a-95a0-e23b8584daaf'}) [timeout_sampler] 2026-09-10T06:57:13.879209Z [info ] Job 3b22e4b8-2ba0-417a-95a0-e23b8584daaf state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:23.893101Z [info ] Job 3b22e4b8-2ba0-417a-95a0-e23b8584daaf state: failed [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:23.894518Z [success ] Function: tests.ai_safety.evalhub.utils.get_job_status succeeded after 10.01389193534851 [0:00:10.013892]. [timeout_sampler] 2026-09-10T06:57:23.894664Z [info ] Waiting for 120 seconds [0:02:00], retry every 1 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._poll) [timeout_sampler] 2026-09-10T06:57:23.958486Z [info ] Failed phase label not read on 3b22e4b8-2ba0-417a-95a0-e2-4b8fef23-593d-439f-982c-c004a8a64c78; operator removed Job after failure sync (label patched before delete) [tests.ai_safety.evalhub.k8s_lifecycle_signals.utils] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.utils 2026-09-10T06:57:23.959638Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._poll succeeded after 7.462501525878906e-05 [0:00:00.000075]. [timeout_sampler] PASSED TEST: TestE2eLifecycle.test_e2e_003_infrastructure_failure_lifecycle STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- _ 3 of 21 completed, 2 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_e2e.py::TestE2eLifecycle::test_e2e_004_threshold_violation_lifecycle  -------------------------------------- test_e2e_004_threshold_violation_lifecycle -------------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-10T06:57:23.996226Z [info ] Submitting evaluation job to https://evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev/api/v1/evaluations/jobs for tenant test-k8s-lifecycle-signals [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:24.028005Z [info ] Job submitted: 67b78cb0-0d60-4cfc-beee-c79218d8eeca [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:24.029313Z [info ] Waiting for 600 seconds [0:10:00], retry every 3 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find) [timeout_sampler] 2026-09-10T06:57:27.062044Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find succeeded after 3.015676498413086 [0:00:03.015676]. [timeout_sampler] 2026-09-10T06:57:27.061291Z [info ] Waiting for job 67b78cb0-0d60-4cfc-beee-c79218d8eeca to complete (timeout=600s) [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:27.062567Z [info ] Waiting for 600 seconds [0:10:00], retry every 10 seconds. (Function: tests.ai_safety.evalhub.utils.get_job_status Kwargs: {'host': 'evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev', 'token': '***', 'ca_bundle_file': '/tmp/pytest-CNwLJSa3c8E2UKj8CtwqYu/tests0/openshift_ca.crt', 'tenant': 'test-k8s-lifecycle-signals', 'job_id': '67b78cb0-0d60-4cfc-beee-c79218d8eeca'}) [timeout_sampler] 2026-09-10T06:57:27.090659Z [info ] Job 67b78cb0-0d60-4cfc-beee-c79218d8eeca state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:37.111896Z [info ] Job 67b78cb0-0d60-4cfc-beee-c79218d8eeca state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:47.142632Z [info ] Job 67b78cb0-0d60-4cfc-beee-c79218d8eeca state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:57.166077Z [info ] Job 67b78cb0-0d60-4cfc-beee-c79218d8eeca state: completed [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:57.167430Z [success ] Function: tests.ai_safety.evalhub.utils.get_job_status succeeded after 30.081513166427612 [0:00:30.081513]. [timeout_sampler] 2026-09-10T06:57:57.167592Z [info ] Waiting for 240 seconds [0:04:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event) [timeout_sampler] 2026-09-10T06:57:57.185466Z [info ] Event EvaluationRunning emitted for job 67b78cb0-0d60-4cfc-beee-c7-c0e454c3-257d-4216-8654-7435ea723d96 (elapsed=0.0s) [tests.ai_safety.evalhub.k8s_lifecycle_signals.utils] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.utils 2026-09-10T06:57:57.186313Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event succeeded after 3.147125244140625e-05 [0:00:00.000031]. [timeout_sampler] 2026-09-10T06:57:57.186460Z [info ] Waiting for 240 seconds [0:04:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event) [timeout_sampler] 2026-09-10T06:57:57.201779Z [info ] Event EvaluationThresholdViolated emitted for job 67b78cb0-0d60-4cfc-beee-c7-c0e454c3-257d-4216-8654-7435ea723d96 (elapsed=0.0s) [tests.ai_safety.evalhub.k8s_lifecycle_signals.utils] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.utils 2026-09-10T06:57:57.202855Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event succeeded after 3.0517578125e-05 [0:00:00.000031]. [timeout_sampler] 2026-09-10T06:57:57.202981Z [info ] Waiting for 120 seconds [0:02:00], retry every 5 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label) [timeout_sampler] 2026-09-10T06:57:57.234674Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label succeeded after 3.218650817871094e-05 [0:00:00.000032]. [timeout_sampler] PASSED _ 4 of 21 completed, 3 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_event_emission.py::TestEvtEventEmission::test_evt_001_evaluation_started_event_emitted  TEST: TestE2eLifecycle.test_e2e_004_threshold_violation_lifecycle STATUS: PASSED ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ------------------------------------ test_evt_001_evaluation_started_event_emitted ------------------------------------ -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-10T06:57:57.305588Z [info ] Submitting evaluation job to https://evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev/api/v1/evaluations/jobs for tenant test-k8s-lifecycle-signals [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:57.378823Z [info ] Job submitted: 162d6392-91f1-494e-9595-b735ef0d472a [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:57:57.379953Z [info ] Waiting for 600 seconds [0:10:00], retry every 3 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find) [timeout_sampler] 2026-09-10T06:58:00.412466Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find succeeded after 3.0155680179595947 [0:00:03.015568]. [timeout_sampler] 2026-09-10T06:58:00.412629Z [info ] Waiting for 240 seconds [0:04:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event) [timeout_sampler] 2026-09-10T06:58:08.495989Z [info ] Event EvaluationRunning emitted for job 162d6392-91f1-494e-9595-b7-767db626-07f2-4b51-9a50-17b7a7e51dd6 (elapsed=8.1s) [tests.ai_safety.evalhub.k8s_lifecycle_signals.utils] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.utils 2026-09-10T06:58:08.497203Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event succeeded after 8.067272901535034 [0:00:08.067273]. [timeout_sampler] PASSED TEST: TestEvtEventEmission.test_evt_001_evaluation_started_event_emitted STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- _ 5 of 21 completed, 4 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_event_emission.py::TestEvtEventEmission::test_evt_002_evaluation_completed_event_emitted  ----------------------------------- test_evt_002_evaluation_completed_event_emitted ----------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-10T06:58:08.532973Z [info ] Submitting evaluation job to https://evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev/api/v1/evaluations/jobs for tenant test-k8s-lifecycle-signals [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:58:08.569321Z [info ] Job submitted: 3ca7be1f-e5f4-4675-9c03-110f3158d408 [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:58:08.570525Z [info ] Waiting for 600 seconds [0:10:00], retry every 3 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find) [timeout_sampler] 2026-09-10T06:58:11.603729Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find succeeded after 3.0159802436828613 [0:00:03.015980]. [timeout_sampler] 2026-09-10T06:58:11.602796Z [info ] Waiting for job 3ca7be1f-e5f4-4675-9c03-110f3158d408 to complete (timeout=600s) [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:58:11.604331Z [info ] Waiting for 600 seconds [0:10:00], retry every 10 seconds. (Function: tests.ai_safety.evalhub.utils.get_job_status Kwargs: {'host': 'evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev', 'token': '***', 'ca_bundle_file': '/tmp/pytest-CNwLJSa3c8E2UKj8CtwqYu/tests0/openshift_ca.crt', 'tenant': 'test-k8s-lifecycle-signals', 'job_id': '3ca7be1f-e5f4-4675-9c03-110f3158d408'}) [timeout_sampler] 2026-09-10T06:58:11.715356Z [info ] Job 3ca7be1f-e5f4-4675-9c03-110f3158d408 state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:58:21.738793Z [info ] Job 3ca7be1f-e5f4-4675-9c03-110f3158d408 state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:58:31.767544Z [info ] Job 3ca7be1f-e5f4-4675-9c03-110f3158d408 state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:58:41.781044Z [info ] Job 3ca7be1f-e5f4-4675-9c03-110f3158d408 state: completed [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:58:41.782327Z [success ] Function: tests.ai_safety.evalhub.utils.get_job_status succeeded after 30.164899349212646 [0:00:30.164899]. [timeout_sampler] 2026-09-10T06:58:41.782526Z [info ] Waiting for 240 seconds [0:04:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event) [timeout_sampler] 2026-09-10T06:58:41.800671Z [info ] Event EvaluationCompleted emitted for job 3ca7be1f-e5f4-4675-9c03-11-6d5f22c5-68c2-47a3-9001-8ab118c7c551 (elapsed=0.0s) [tests.ai_safety.evalhub.k8s_lifecycle_signals.utils] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.utils 2026-09-10T06:58:41.801683Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event succeeded after 3.123283386230469e-05 [0:00:00.000031]. [timeout_sampler] PASSED TEST: TestEvtEventEmission.test_evt_002_evaluation_completed_event_emitted STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- _ 6 of 21 completed, 5 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_event_emission.py::TestEvtEventEmission::test_evt_003_evaluation_failed_event_emitted_on_adapter_error  ---------------------------- test_evt_003_evaluation_failed_event_emitted_on_adapter_error ---------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-10T06:58:41.838141Z [info ] Submitting evaluation job to https://evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev/api/v1/evaluations/jobs for tenant test-k8s-lifecycle-signals [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:58:41.881882Z [info ] Job submitted: ba44ae05-1aab-4876-a495-944399cf689b [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:58:41.883227Z [info ] Waiting for 600 seconds [0:10:00], retry every 3 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find) [timeout_sampler] 2026-09-10T06:58:44.916092Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find succeeded after 3.0157034397125244 [0:00:03.015703]. [timeout_sampler] 2026-09-10T06:58:44.915179Z [info ] Waiting for job ba44ae05-1aab-4876-a495-944399cf689b to complete (timeout=600s) [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:58:44.916492Z [info ] Waiting for 600 seconds [0:10:00], retry every 10 seconds. (Function: tests.ai_safety.evalhub.utils.get_job_status Kwargs: {'host': 'evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev', 'token': '***', 'ca_bundle_file': '/tmp/pytest-CNwLJSa3c8E2UKj8CtwqYu/tests0/openshift_ca.crt', 'tenant': 'test-k8s-lifecycle-signals', 'job_id': 'ba44ae05-1aab-4876-a495-944399cf689b'}) [timeout_sampler] 2026-09-10T06:58:44.927175Z [info ] Job ba44ae05-1aab-4876-a495-944399cf689b state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:58:54.965166Z [info ] Job ba44ae05-1aab-4876-a495-944399cf689b state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:59:04.978305Z [info ] Job ba44ae05-1aab-4876-a495-944399cf689b state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:59:14.992219Z [info ] Job ba44ae05-1aab-4876-a495-944399cf689b state: failed [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:59:14.993612Z [success ] Function: tests.ai_safety.evalhub.utils.get_job_status succeeded after 30.063270330429077 [0:00:30.063270]. [timeout_sampler] 2026-09-10T06:59:14.993795Z [info ] Waiting for 240 seconds [0:04:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event) [timeout_sampler] 2026-09-10T06:59:15.012972Z [info ] Event EvaluationFailed emitted for job ba44ae05-1aab-4876-a495-94-d6098123-d17f-4f94-ba2a-7ae4ae8c459c (elapsed=0.0s) [tests.ai_safety.evalhub.k8s_lifecycle_signals.utils] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.utils 2026-09-10T06:59:15.014119Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event succeeded after 3.4809112548828125e-05 [0:00:00.000035]. [timeout_sampler] PASSED TEST: TestEvtEventEmission.test_evt_003_evaluation_failed_event_emitted_on_adapter_error STATUS: PASSED ------------------------------------------------------- TEARDOWN ------------------------------------------------------- _ 7 of 21 completed, 6 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_event_emission.py::TestEvtEventEmission::test_evt_004_evaluation_threshold_violated_event_emitted  ------------------------------- test_evt_004_evaluation_threshold_violated_event_emitted ------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-10T06:59:15.051734Z [info ] Submitting evaluation job to https://evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev/api/v1/evaluations/jobs for tenant test-k8s-lifecycle-signals [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:59:15.086611Z [info ] Job submitted: cbf0ad5b-1c72-4e7b-987c-1a511c054700 [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:59:15.087624Z [info ] Waiting for 600 seconds [0:10:00], retry every 3 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find) [timeout_sampler] 2026-09-10T06:59:18.121794Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find succeeded after 3.015789031982422 [0:00:03.015789]. [timeout_sampler] 2026-09-10T06:59:18.120801Z [info ] Waiting for job cbf0ad5b-1c72-4e7b-987c-1a511c054700 to complete (timeout=600s) [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:59:18.122234Z [info ] Waiting for 600 seconds [0:10:00], retry every 10 seconds. (Function: tests.ai_safety.evalhub.utils.get_job_status Kwargs: {'host': 'evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev', 'token': '***', 'ca_bundle_file': '/tmp/pytest-CNwLJSa3c8E2UKj8CtwqYu/tests0/openshift_ca.crt', 'tenant': 'test-k8s-lifecycle-signals', 'job_id': 'cbf0ad5b-1c72-4e7b-987c-1a511c054700'}) [timeout_sampler] 2026-09-10T06:59:18.137684Z [info ] Job cbf0ad5b-1c72-4e7b-987c-1a511c054700 state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:59:28.153825Z [info ] Job cbf0ad5b-1c72-4e7b-987c-1a511c054700 state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:59:38.226844Z [info ] Job cbf0ad5b-1c72-4e7b-987c-1a511c054700 state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:59:48.252770Z [info ] Job cbf0ad5b-1c72-4e7b-987c-1a511c054700 state: completed [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:59:48.254273Z [success ] Function: tests.ai_safety.evalhub.utils.get_job_status succeeded after 30.106853246688843 [0:00:30.106853]. [timeout_sampler] 2026-09-10T06:59:48.254429Z [info ] Waiting for 240 seconds [0:04:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event) [timeout_sampler] 2026-09-10T06:59:48.272889Z [info ] Event EvaluationThresholdViolated emitted for job cbf0ad5b-1c72-4e7b-987c-1a-b922f2ab-3687-4096-a5b5-324f795dac08 (elapsed=0.0s) [tests.ai_safety.evalhub.k8s_lifecycle_signals.utils] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.utils 2026-09-10T06:59:48.274428Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event succeeded after 4.7206878662109375e-05 [0:00:00.000047]. [timeout_sampler] PASSED TEST: TestEvtEventEmission.test_evt_004_evaluation_threshold_violated_event_emitted STATUS: PASSED _ 8 of 21 completed, 7 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_job_label_phase.py::TestLblJobLabelPhase::test_lbl_001_job_label_set_to_running_on_start ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ------------------------------------ test_lbl_001_job_label_set_to_running_on_start ------------------------------------ -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-10T06:59:48.312465Z [info ] Submitting evaluation job to https://evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev/api/v1/evaluations/jobs for tenant test-k8s-lifecycle-signals [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:59:48.344432Z [info ] Job submitted: 8f4787df-a62e-4538-842b-8e09246c35f5 [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T06:59:48.345939Z [info ] Waiting for 600 seconds [0:10:00], retry every 3 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find) [timeout_sampler] 2026-09-10T06:59:51.378055Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find succeeded after 3.015709638595581 [0:00:03.015710]. [timeout_sampler] 2026-09-10T06:59:51.378236Z [info ] Waiting for 60 seconds [0:01:00], retry every 5 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label) [timeout_sampler] PASSED2026-09-10T07:00:01.473815Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label succeeded after 10.062883615493774 [0:00:10.062884]. [timeout_sampler] _ 9 of 21 completed, 8 Pass, 1 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_job_label_phase.py::TestLblJobLabelPhase::test_lbl_002_job_label_set_to_succeeded_on_completion  TEST: TestLblJobLabelPhase.test_lbl_001_job_label_set_to_running_on_start STATUS: PASSED ------------------------------------------------------- TEARDOWN -------------------------------------------------------  -------------------------------- test_lbl_002_job_label_set_to_succeeded_on_completion -------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-10T07:00:01.493854Z [info ] Submitting evaluation job to https://evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev/api/v1/evaluations/jobs for tenant test-k8s-lifecycle-signals [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:00:01.583172Z [info ] Job submitted: f1d6208a-375a-4a32-a3f9-77c83538c5e6 [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:00:01.584378Z [info ] Waiting for 600 seconds [0:10:00], retry every 3 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find) [timeout_sampler] 2026-09-10T07:00:04.616499Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find succeeded after 3.015587091445923 [0:00:03.015587]. [timeout_sampler] 2026-09-10T07:00:04.615748Z [info ] Waiting for job f1d6208a-375a-4a32-a3f9-77c83538c5e6 to complete (timeout=600s) [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:00:04.616858Z [info ] Waiting for 600 seconds [0:10:00], retry every 10 seconds. (Function: tests.ai_safety.evalhub.utils.get_job_status Kwargs: {'host': 'evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev', 'token': '***', 'ca_bundle_file': '/tmp/pytest-CNwLJSa3c8E2UKj8CtwqYu/tests0/openshift_ca.crt', 'tenant': 'test-k8s-lifecycle-signals', 'job_id': 'f1d6208a-375a-4a32-a3f9-77c83538c5e6'}) [timeout_sampler] 2026-09-10T07:00:04.717648Z [info ] Job f1d6208a-375a-4a32-a3f9-77c83538c5e6 state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:00:14.735437Z [info ] Job f1d6208a-375a-4a32-a3f9-77c83538c5e6 state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:00:24.754906Z [info ] Job f1d6208a-375a-4a32-a3f9-77c83538c5e6 state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:00:34.775873Z [info ] Job f1d6208a-375a-4a32-a3f9-77c83538c5e6 state: completed [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:00:34.777329Z [success ] Function: tests.ai_safety.evalhub.utils.get_job_status succeeded after 30.139288425445557 [0:00:30.139288]. [timeout_sampler] 2026-09-10T07:00:34.777587Z [info ] Waiting for 240 seconds [0:04:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event) [timeout_sampler] 2026-09-10T07:00:34.796444Z [info ] Event EvaluationCompleted emitted for job f1d6208a-375a-4a32-a3f9-77-4424742f-6411-4f5b-b445-74ce3a9c2156 (elapsed=0.0s) [tests.ai_safety.evalhub.k8s_lifecycle_signals.utils] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.utils 2026-09-10T07:00:34.797269Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event succeeded after 3.337860107421875e-05 [0:00:00.000033]. [timeout_sampler] 2026-09-10T07:00:34.797417Z [info ] Waiting for 120 seconds [0:02:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label) [timeout_sampler] 2026-09-10T07:02:00.327713Z [info ] Delete Service vllm-emulator-service [ocp_resources Service] 2026-09-10T07:02:00.359929Z [info ] Deleting {'kind': 'Service', 'apiVersion': 'v1', 'metadata': {'name': 'vllm-emulator-service', 'namespace': 'test-k8s-lifecycle-signals', 'uid': 'dad29420-05f6-4368-8a16-a3535b85eb2c', 'resourceVersion': '115398', 'creationTimestamp': '2026-09-10T06:52:42Z', 'managedFields': [{'manager': 'OpenAPI-Generator', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2026-09-10T06:52:42Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:internalTrafficPolicy': {}, 'f:ports': {'.': {}, 'k:{"port":8000,"protocol":"TCP"}': {'.': {}, 'f:name': {}, 'f:port': {}, 'f:protocol': {}, 'f:targetPort': {}}}, 'f:selector': {}, 'f:sessionAffinity': {}, 'f:type': {}}}}]}, 'spec': {'ports': [{'name': 'vllm-emulator-endpoint', 'protocol': 'TCP', 'port': 8000, 'targetPort': 8000}], 'selector': {'app': 'vllm-emulator'}, 'clusterIP': '172.31.100.45', 'clusterIPs': ['172.31.100.45'], 'type': 'ClusterIP', 'sessionAffinity': 'None', 'ipFamilies': ['IPv4'], 'ipFamilyPolicy': 'SingleStack', 'internalTrafficPolicy': 'Cluster'}, 'status': {'loadBalancer': {}}} [ocp_resources Service] 2026-09-10T07:02:00.383861Z [info ] Wait until Service vllm-emulator-service is deleted [ocp_resources Service] 2026-09-10T07:02:00.384048Z [info ] Waiting for 240 seconds [0:04:00], retry every 1 seconds. (Function: ocp_resources.resource.wait_deleted.lambda: self.exists) [timeout_sampler] 2026-09-10T07:02:00.401073Z [success ] Function: lambda: self.exists succeeded after 6.461143493652344e-05 [0:00:00.000065]. [timeout_sampler] 2026-09-10T07:02:00.401313Z [error ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label failed after 85.3585479259491 [0:01:25.358548]. [timeout_sampler] FAILED __ TestLblJobLabelPhase.test_lbl_002_job_label_set_to_succeeded_on_completion __ Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_job_label_phase.py", line 136, in test_lbl_002_job_label_set_to_succeeded_on_completion wait_for_success_phase_signals( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ admin_client=admin_client, ^^^^^^^^^^^^^^^^^^^^^^^^^^ job_name=job_name, ^^^^^^^^^^^^^^^^^^ namespace=ns, ^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/k8s_lifecycle_signals/utils.py", line 154, in wait_for_success_phase_signals wait_for_job_label( ~~~~~~~~~~~~~~~~~~^ admin_client=admin_client, ^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<5 lines>... sleep=2, ^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/k8s_lifecycle_signals/utils.py", line 126, in wait_for_job_label for value in TimeoutSampler(wait_timeout=timeout, sleep=sleep, func=_get_label): ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 319, in __iter__ time.sleep(self.sleep) ~~~~~~~~~~^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 800, in _sigint_handler sys.exit(signal_received) ~~~~~~~~^^^^^^^^^^^^^^^^^ SystemExit: 2 _ 10 of 21 completed, 8 Pass, 2 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_job_label_phase.py::TestLblJobLabelPhase::test_lbl_003_job_label_set_to_failed_on_failure  TEST: TestLblJobLabelPhase.test_lbl_002_job_label_set_to_succeeded_on_completion STATUS: FAILED 2026-09-10T07:02:00.407086Z [error ] Traceback (most recent call last): File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 361, in from_call result: TResult | None = func() ~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 250, in lambda: runtest_hook(item=item, **kwds), ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call return (yield) ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/runner.py", line 184, in pytest_runtest_call item.runtest() ~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 1707, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall raise exception File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall teardown.throw(exception) ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper return result.get_result() ~~~~~~~~~~~~~~~~~^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_result.py", line 103, in get_result raise exc.with_traceback(tb) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper res = yield ^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall res = hook_impl.function(*args) File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/_pytest/python.py", line 167, in pytest_pyfunc_call result = testfunction(**testargs) File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_job_label_phase.py", line 136, in test_lbl_002_job_label_set_to_succeeded_on_completion wait_for_success_phase_signals( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ admin_client=admin_client, ^^^^^^^^^^^^^^^^^^^^^^^^^^ job_name=job_name, ^^^^^^^^^^^^^^^^^^ namespace=ns, ^^^^^^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/k8s_lifecycle_signals/utils.py", line 154, in wait_for_success_phase_signals wait_for_job_label( ~~~~~~~~~~~~~~~~~~^ admin_client=admin_client, ^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<5 lines>... sleep=2, ^^^^^^^^ ) ^ File "/home/odh/opendatahub-tests/tests/ai_safety/evalhub/k8s_lifecycle_signals/utils.py", line 126, in wait_for_job_label for value in TimeoutSampler(wait_timeout=timeout, sleep=sleep, func=_get_label): ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/timeout_sampler/__init__.py", line 319, in __iter__ time.sleep(self.sleep) ~~~~~~~~~~^^^^^^^^^^^^ File "/home/odh/opendatahub-tests/.venv/lib64/python3.14/site-packages/ocp_resources/resource.py", line 800, in _sigint_handler sys.exit(signal_received) ~~~~~~~~^^^^^^^^^^^^^^^^^ SystemExit: 2 [conftest] ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ----------------------------------- test_lbl_003_job_label_set_to_failed_on_failure ----------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-10T07:02:00.425917Z [info ] Submitting evaluation job to https://evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev/api/v1/evaluations/jobs for tenant test-k8s-lifecycle-signals [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:02:00.603604Z [info ] Job submitted: 9b69c871-c647-44ac-9d05-31c12fc45b28 [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:02:00.605165Z [info ] Waiting for 600 seconds [0:10:00], retry every 3 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find) [timeout_sampler] 2026-09-10T07:02:03.639303Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find succeeded after 3.0158889293670654 [0:00:03.015889]. [timeout_sampler] 2026-09-10T07:02:03.637696Z [info ] Waiting for job 9b69c871-c647-44ac-9d05-31c12fc45b28 to complete (timeout=600s) [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:02:03.639988Z [info ] Waiting for 600 seconds [0:10:00], retry every 10 seconds. (Function: tests.ai_safety.evalhub.utils.get_job_status Kwargs: {'host': 'evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev', 'token': '***', 'ca_bundle_file': '/tmp/pytest-CNwLJSa3c8E2UKj8CtwqYu/tests0/openshift_ca.crt', 'tenant': 'test-k8s-lifecycle-signals', 'job_id': '9b69c871-c647-44ac-9d05-31c12fc45b28'}) [timeout_sampler] 2026-09-10T07:02:03.668090Z [info ] Job 9b69c871-c647-44ac-9d05-31c12fc45b28 state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:02:13.778016Z [info ] Job 9b69c871-c647-44ac-9d05-31c12fc45b28 state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:02:23.804753Z [info ] Job 9b69c871-c647-44ac-9d05-31c12fc45b28 state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:02:33.872880Z [info ] Job 9b69c871-c647-44ac-9d05-31c12fc45b28 state: failed [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:02:33.874264Z [success ] Function: tests.ai_safety.evalhub.utils.get_job_status succeeded after 30.16717267036438 [0:00:30.167173]. [timeout_sampler] 2026-09-10T07:02:33.874500Z [info ] Waiting for 240 seconds [0:04:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event) [timeout_sampler] 2026-09-10T07:02:33.894262Z [info ] Event EvaluationFailed emitted for job 9b69c871-c647-44ac-9d05-31-195194c7-89a6-4b8b-9aa5-a884d64894db (elapsed=0.0s) [tests.ai_safety.evalhub.k8s_lifecycle_signals.utils] name=tests.ai_safety.evalhub.k8s_lifecycle_signals.utils 2026-09-10T07:02:33.895557Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find_event succeeded after 8.869171142578125e-05 [0:00:00.000089]. [timeout_sampler] 2026-09-10T07:02:33.895752Z [info ] Waiting for 120 seconds [0:02:00], retry every 2 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label) [timeout_sampler] 2026-09-10T07:02:33.927299Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._get_label succeeded after 4.100799560546875e-05 [0:00:00.000041]. [timeout_sampler] PASSED _ 11 of 21 completed, 9 Pass, 2 Fail, 0 Skip, 0 XPass, 0 XFail, 0 Error, 0 ReRun _ tests/ai_safety/evalhub/k8s_lifecycle_signals/test_evalhub_lifecycle_job_label_phase.py::TestLblJobLabelPhase::test_lbl_004_job_label_set_to_threshold_violated  TEST: TestLblJobLabelPhase.test_lbl_003_job_label_set_to_failed_on_failure STATUS: PASSED ------------------------------------------------------- TEARDOWN -------------------------------------------------------  ----------------------------------- test_lbl_004_job_label_set_to_threshold_violated ----------------------------------- -------------------------------------------------------- SETUP -------------------------------------------------------- --------------------------------------------------------- CALL --------------------------------------------------------- 2026-09-10T07:02:33.979476Z [info ] Submitting evaluation job to https://evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev/api/v1/evaluations/jobs for tenant test-k8s-lifecycle-signals [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:02:34.010193Z [info ] Job submitted: 670c7240-12ba-44f8-8e91-45f76f75d7ec [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:02:34.011665Z [info ] Waiting for 600 seconds [0:10:00], retry every 3 seconds. (Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find) [timeout_sampler] 2026-09-10T07:02:37.044293Z [success ] Function: tests.ai_safety.evalhub.k8s_lifecycle_signals.utils._find succeeded after 3.0156850814819336 [0:00:03.015685]. [timeout_sampler] 2026-09-10T07:02:37.043232Z [info ] Waiting for job 670c7240-12ba-44f8-8e91-45f76f75d7ec to complete (timeout=600s) [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:02:37.044820Z [info ] Waiting for 600 seconds [0:10:00], retry every 10 seconds. (Function: tests.ai_safety.evalhub.utils.get_job_status Kwargs: {'host': 'evalhub-ls-test-k8s-lifecycle-signals-cp.apps.de1913e7c77c35f73c2d.prod.konflux-ocp-ci.dev', 'token': '***', 'ca_bundle_file': '/tmp/pytest-CNwLJSa3c8E2UKj8CtwqYu/tests0/openshift_ca.crt', 'tenant': 'test-k8s-lifecycle-signals', 'job_id': '670c7240-12ba-44f8-8e91-45f76f75d7ec'}) [timeout_sampler] 2026-09-10T07:02:37.062084Z [info ] Job 670c7240-12ba-44f8-8e91-45f76f75d7ec state: pending [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:02:47.175361Z [info ] Job 670c7240-12ba-44f8-8e91-45f76f75d7ec state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils 2026-09-10T07:02:57.190671Z [info ] Job 670c7240-12ba-44f8-8e91-45f76f75d7ec state: running [tests.ai_safety.evalhub.utils] name=tests.ai_safety.evalhub.utils