Fetching https://github.com/opendatahub-io/models-as-a-service@ccce7752af146eb3de9e99df107c26f414878726 into /workspace/source/test/maas-e2e ... MaaS checkout at: ccce7752af146eb3de9e99df107c26f414878726 Patched /workspace/source/test/maas-e2e/scripts/deploy.sh Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS checkout fixture root) Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS AuthPolicy wait scope) NOTE: rewriting AI_GATEWAY_CONTROLLER_IMAGE from snapshot digest to tag odh-pr-184 MaaS platform images: MAAS_API_IMAGE=quay.io/opendatahub/maas-api:ccce7752af146eb3de9e99df107c26f414878726 MAAS_CONTROLLER_IMAGE=quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726 AI_GATEWAY_CONTROLLER_IMAGE=quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-184 PRAXIS_EXTPROC_IMAGE=quay.io/opendatahub/odh-praxis-extproc:odh-stable ---------------------------------------- ai-gateway-controller E2E on OpenShift ---------------------------------------- Checking prerequisites... Prerequisites met — logged in as: system:admin DEPLOY_MODE: kustomize MAAS_API_IMAGE: quay.io/opendatahub/maas-api:ccce7752af146eb3de9e99df107c26f414878726 MAAS_CONTROLLER_IMAGE: quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726 AI_GATEWAY_CONTROLLER_IMAGE: quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-184 PRAXIS_EXTPROC_IMAGE: quay.io/opendatahub/odh-praxis-extproc:odh-stable 2026-10-30T18:40:00Z deploy_platform start Deploying MaaS platform via ODH operator... Gateway ingress mode for deploy.sh: ocproute (loadbalancer | ocproute) Using custom MaaS API image: quay.io/opendatahub/maas-api:ccce7752af146eb3de9e99df107c26f414878726 Using custom MaaS controller image: quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726 Deployment mode: kustomize Installing cert-manager and LeaderWorkerSet operators... === Installing cert-manager and LeaderWorkerSet operators === 1. Installing cert-manager operator... namespace/cert-manager-operator created operatorgroup.operators.coreos.com/cert-manager-operator created subscription.operators.coreos.com/openshift-cert-manager-operator created Waiting for Subscription cert-manager-operator/openshift-cert-manager-operator... subscription.operators.coreos.com/openshift-cert-manager-operator condition met Waiting for CSV cert-manager-operator.v1.20.1 to succeed... clusterserviceversion.operators.coreos.com/cert-manager-operator.v1.20.1 condition met cert-manager ready. 2. Installing LeaderWorkerSet operator... namespace/openshift-lws-operator created operatorgroup.operators.coreos.com/leader-worker-set created subscription.operators.coreos.com/leader-worker-set created Waiting for Subscription openshift-lws-operator/leader-worker-set... subscription.operators.coreos.com/leader-worker-set condition met Waiting for CSV leader-worker-set.v1.0.1 to succeed... clusterserviceversion.operators.coreos.com/leader-worker-set.v1.0.1 condition met LeaderWorkerSet operator ready. 3. Activating LeaderWorkerSet API... leaderworkersetoperator.operator.openshift.io/cluster created LeaderWorkerSetOperator CR applied. === Done === Verify: oc get pods -n cert-manager-operator oc get pods -n openshift-lws-operator oc get crd leaderworkersets.leaderworkerset.x-k8s.io Installing OpenDataHub operator... === Installing OpenDataHub operator === 1. Setting up ODH catalog... Using community-operators 2. Installing ODH operator... [INFO] Installing operator: opendatahub-operator in namespace: opendatahub [INFO] Creating namespace: opendatahub namespace/opendatahub created namespace/opendatahub condition met [INFO] Creating OperatorGroup in opendatahub for AllNamespaces mode operatorgroup.operators.coreos.com/opendatahub-operatorgroup created [INFO] Creating Subscription for opendatahub-operator from community-operators (channel: fast-3, installPlanApproval: Manual, startingCSV: opendatahub-operator.v3.6.0-ea.2) subscription.operators.coreos.com/opendatahub-operator created [INFO] Manual Subscription: approving initial InstallPlan so first install can proceed... [INFO] Approving initial InstallPlan install-f6n9c (Manual subscription) installplan.operators.coreos.com/install-f6n9c patched [INFO] Waiting for subscription to install... * Waiting for Subscription opendatahub/opendatahub-operator to start setup... subscription.operators.coreos.com/opendatahub-operator condition met * Waiting for Subscription setup to finish setup. CSV = opendatahub-operator.v3.6.0-ea.2 ... clusterserviceversion.operators.coreos.com/opendatahub-operator.v3.6.0-ea.2 condition met [INFO] Operator opendatahub-operator installed successfully 3. Skipping operator image patch (OPERATOR_IMAGE not set) 4. Waiting for operator CRDs... ⏳ Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 180s)… ✅ CRD datascienceclusters.datasciencecluster.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met 5. Waiting for operator webhook... * Waiting for deployment/opendatahub-operator-controller-manager in opendatahub (timeout: 120s)... * Found deployment/opendatahub-operator-controller-manager deployment.apps/opendatahub-operator-controller-manager condition met 6. Applying DSCInitialization... dscinitialization.dscinitialization.opendatahub.io/default-dsci created Waiting for DSCInitialization to be Ready... Waiting for DSCInitialization Ready (attempt 1/30, phase=unknown, Ready=unknown)... DSCInitialization is Ready 7. Applying DataScienceCluster... datasciencecluster.datasciencecluster.opendatahub.io/default-dsc serverside-applied 8. Waiting for DataScienceCluster (KServe)... * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... - KServe state: , KserveReady: , ModelsAsServiceReady: (informational only), AIGatewayReady: - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: (informational only), AIGatewayReady: False - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: (informational only), AIGatewayReady: False - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: (informational only), AIGatewayReady: False - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: (informational only), AIGatewayReady: False - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: (informational only), AIGatewayReady: False - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: (informational only), AIGatewayReady: False * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' 9. Waiting for odh-model-controller webhook... ⏳ Waiting for validating webhooks in namespace opendatahub (timeout: 180s)... ✅ Webhook service opendatahub/kserve-webhook-server-service has ready endpoints ✅ Webhook service opendatahub/llmisvc-webhook-server-service has ready endpoints ✅ Webhook service opendatahub/odh-model-controller-webhook-service has ready endpoints ✅ Webhook service opendatahub/opendatahub-operator-controller-manager-service has ready endpoints 🎉 All validating webhook services in opendatahub are ready === ODH installation complete === Verify: kubectl get datasciencecluster -A kubectl get pods -n opendatahub kubectl get pods -n kserve Using policy engine: rhcl (Authorino namespace: kuadrant-system) [INFO] =================================================== [INFO] Models-as-a-Service Deployment [INFO] =================================================== [INFO] Validating configuration... [INFO] Configuration validated successfully [INFO] Deployment configuration: [INFO] Mode: kustomize [INFO] Policy Engine: rhcl [INFO] Namespace: opendatahub [INFO] TLS Backend: true [INFO] External OIDC: false [INFO] MaaS API image: quay.io/opendatahub/maas-api:ccce7752af146eb3de9e99df107c26f414878726 [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726 [INFO] Starting kustomize-based deployment... [INFO] Installing policy engine: rhcl [INFO] Installing RHCL (Red Hat Connectivity Link - downstream) [INFO] Using RHCL channel head from redhat-operators (stable) [INFO] Installing RHCL into namespace: kuadrant-system [INFO] Installing operator: rhcl-operator in namespace: kuadrant-system [INFO] Creating namespace: kuadrant-system namespace/kuadrant-system created namespace/kuadrant-system condition met [INFO] Creating OperatorGroup in kuadrant-system for AllNamespaces mode operatorgroup.operators.coreos.com/kuadrant-system-operatorgroup created [INFO] Creating Subscription for rhcl-operator from redhat-operators (channel: stable) subscription.operators.coreos.com/rhcl-operator created [INFO] Waiting for subscription to install... * Waiting for Subscription kuadrant-system/rhcl-operator to start setup... subscription.operators.coreos.com/rhcl-operator condition met * Waiting for Subscription setup to finish setup. CSV = rhcl-operator.v1.4.3 ... clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 condition met [INFO] Operator rhcl-operator installed successfully [INFO] Patching rhcl-operator CSV (Gateway API, rate limit failure modes, auth service timeout, resources)... clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched [INFO] CSV patched (Gateway controller and/or rate limit failure modes and/or auth timeout and/or resources) [INFO] Forcing operator restart to apply CSV env configuration... pod "kuadrant-operator-controller-manager-6b6fb56846-847vz" force deleted pod "kuadrant-operator-controller-manager-6b76784d48-8rhnm" force deleted pod "limitador-operator-controller-manager-856dd584c5-m9qzq" force deleted [INFO] Waiting for operator pod to restart... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... deployment "kuadrant-operator-controller-manager" successfully rolled out [WARN] Operator pod may not have correct env yet (ISTIO / RATELIMIT_* failure modes / AUTH_SERVICE_TIMEOUT) [INFO] Operator deployment has memory limit 1Gi [INFO] Waiting 15s for operator to fully initialize with Gateway controller configuration... [INFO] Initializing Gateway API and ModelsAsAService gateway... [INFO] =================================================== [INFO] MaaS Gateway Setup [INFO] =================================================== [INFO] Validating gateway configuration... [INFO] Configuration validated [INFO] Ingress mode: ocproute [INFO] Disconnected: false [INFO] AllowedRoutes: namespaces=opendatahub,odh-ai-gateway-infra,llm [INFO] Detecting cluster domain... [INFO] Detected cluster domain: apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev [INFO] Setting up GatewayClass... [INFO] Creating GatewayClass openshift-default... gatewayclass.gateway.networking.k8s.io/openshift-default created [INFO] Setting up Gateway in ocproute mode (ClusterIP with OpenShift Route)... [INFO] Creating ConfigMap gw-options... configmap/gw-options created [INFO] Creating/updating Gateway maas-default-gateway (ocproute mode)... gateway.gateway.networking.k8s.io/maas-default-gateway serverside-applied [INFO] Waiting for Gateway to be Programmed (timeout: 120s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met [INFO] Gateway is Programmed [INFO] Gateway Service ready: maas-default-gateway-openshift-default [INFO] Creating Route maas-gateway-route... [INFO] Host: maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev [INFO] Target Service: maas-default-gateway-openshift-default route.route.openshift.io/maas-gateway-route created [INFO] Waiting for Route to be Admitted... [INFO] Route is Admitted [INFO] ClusterIP mode setup complete [INFO] [INFO] =================================================== [INFO] Gateway setup completed successfully [INFO] =================================================== [INFO] Mode: ocproute [INFO] Gateway: maas-default-gateway [INFO] Namespace: openshift-ingress [INFO] Route: maas-gateway-route [INFO] Hostname: maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev [INFO] [INFO] Verify with: [INFO] kubectl get gateway maas-default-gateway -n openshift-ingress [INFO] kubectl get route maas-gateway-route -n openshift-ingress [INFO] Applying Kuadrant custom resource in kuadrant-system... kuadrant.kuadrant.io/kuadrant created [INFO] Waiting for Kuadrant to become ready (initial check)... [INFO] Waiting for: Kuadrant ready in kuadrant-system (timeout: 60s) [INFO] Kuadrant ready in kuadrant-system - Ready [INFO] Kuadrant setup complete [INFO] Ensuring namespace exists: opendatahub [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [WARN] DEPLOYING POC POSTGRESQL — NOT INTENDED FOR PRODUCTION USE [WARN] Data is stored in ephemeral storage and will be lost on pod restart. [WARN] For production, use --postgres-connection with an external database [WARN] (AWS RDS, Crunchy Operator, Azure Database, etc.) [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ⚠️ WARNING FOR PRODUCTION USE. ⚠️ ┃ ┃ This deploys PostgreSQL with ephemeral storage (emptyDir). ┃ ┃ Data WILL be lost on pod restart. ┃ ┃ For production, use an external database: ┃ ┃ deploy.sh --postgres-connection postgresql://... ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 🔧 Fresh install: Deploying PostgreSQL in infrastructure namespace 'odh-ai-gateway-infra'... 📦 Creating infrastructure namespace 'odh-ai-gateway-infra'... namespace/odh-ai-gateway-infra created Generated random PostgreSQL password (stored in secret postgres-creds) Creating PostgreSQL deployment... ⚠️ Using POC configuration (ephemeral storage) Using default PostgreSQL image (operator CSV not available) Image: registry.redhat.io/rhel9/postgresql-16:latest secret/postgres-creds created deployment.apps/postgres created service/postgres created secret/maas-db-config created Waiting for PostgreSQL to be ready... deployment.apps/postgres condition met ✅ PostgreSQL deployed successfully Namespace: odh-ai-gateway-infra Database: maas User: maas Secret: maas-db-config (contains DB_CONNECTION_URL with FQDN) ⚠️ For production, use AWS RDS, Crunchy Operator, or Azure Database Note: Schema migrations run automatically when maas-api starts [INFO] Configuring TLS backend for Authorino and MaaS API... * Waiting for deployment/authorino in kuadrant-system (timeout: 300s)... * Found deployment/authorino [INFO] Running TLS configuration script... [INFO] TLS configuration script completed successfully [INFO] Restarting deployments to pick up TLS configuration... deployment.apps/authorino restarted [INFO] Waiting for Authorino deployment to be ready... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment spec update to be observed... Waiting for deployment spec update to be observed... Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out [INFO] TLS backend configuration complete [INFO] Kustomize prerequisite deployment completed [INFO] [INFO] MaaS Controller... [INFO] Phase 1: Applying MaaS CRDs and waiting until Established (controller creates Config after CRD is ready)... ⏳ Applying MaaS Controller CRDs from /workspace/source/test/maas-e2e/scripts/../deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io created ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] Phase 2: Applying full controller kustomize (same as operator: deployment/base/maas-controller/default)... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io configured serviceaccount/maas-controller created role.rbac.authorization.k8s.io/maas-controller-leader-election-role created clusterrole.rbac.authorization.k8s.io/maas-controller-cluster-config-role created clusterrole.rbac.authorization.k8s.io/maas-controller-ocp-role created clusterrole.rbac.authorization.k8s.io/maas-controller-role created clusterrole.rbac.authorization.k8s.io/maas-owner-role created clusterrole.rbac.authorization.k8s.io/maas-viewer-role created rolebinding.rbac.authorization.k8s.io/maas-controller-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-cluster-config-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-ocp-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-rolebinding created configmap/maas-parameters created service/maas-controller-metrics created service/maas-controller-webhook-service created deployment.apps/maas-controller created servicemonitor.monitoring.coreos.com/maas-controller-metrics created networkpolicy.networking.k8s.io/maas-controller-allow-monitoring created validatingwebhookconfiguration.admissionregistration.k8s.io/maas-validating-webhook-configuration created [INFO] Restarting maas-controller to pick up configuration changes... deployment.apps/maas-controller restarted [INFO] Waiting for maas-controller to be ready... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... deployment "maas-controller" successfully rolled out [INFO] Controller ready. [INFO] [INFO] Waiting for Tenant reconciler to deploy maas-api... [INFO] Applying secret migration RBAC to namespace odh-ai-gateway-infra... role.rbac.authorization.k8s.io/maas-controller-secret-migrate created rolebinding.rbac.authorization.k8s.io/maas-controller-secret-migrate created [INFO] maas-api deployment found in odh-ai-gateway-infra, waiting for rollout... Waiting for deployment "maas-api" rollout to finish: 0 of 1 updated replicas are available... deployment "maas-api" successfully rolled out [INFO] maas-api is ready [INFO] [INFO] MaaS API and MaaS Controller deployment completed successfully! [INFO] maas-api image: quay.io/opendatahub/maas-api:ccce7752af146eb3de9e99df107c26f414878726 (namespace: odh-ai-gateway-infra) [INFO] maas-controller image: quay.io/opendatahub/maas-controller:ccce7752af146eb3de9e99df107c26f414878726 (namespace: opendatahub) [INFO] =================================================== [INFO] Models-as-a-Service Deployment completed successfully! [INFO] =================================================== * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' Raising llmisvc-controller-manager CPU in opendatahub (request 1, limit 2)... deployment.apps/llmisvc-controller-manager resource requirements updated Waiting for deployment "llmisvc-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "llmisvc-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "llmisvc-controller-manager" rollout to finish: 1 old replicas are pending termination... deployment "llmisvc-controller-manager" successfully rolled out llmisvc-controller-manager resources: {"limits":{"cpu":"2","memory":"300Mi"},"requests":{"cpu":"1","memory":"300Mi"}} ⚠️ WARNING: Skipping Authorino readiness check (SKIP_AUTH_CHECK=true) ✅ MaaS platform deployment completed 2026-10-30T18:48:08Z deploy_platform end ---------------------------------------- Deploying Models ---------------------------------------- 2026-10-30T18:48:08Z deploy_models start Deploying MaaS system (free + premium: LLMIS + MaaSModelRef + MaaSAuthPolicy + MaaSSubscription) Waiting for Gateway openshift-ingress/maas-default-gateway to be Programmed=True (timeout: 600s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met ✅ Gateway openshift-ingress/maas-default-gateway is Programmed Creating 'llm' namespace... namespace/llm created 'models-as-a-service' namespace already exists maasauthpolicy.maas.opendatahub.io/premium-simulator-access created maasauthpolicy.maas.opendatahub.io/simulator-access created maasmodelref.maas.opendatahub.io/e2e-distinct-2-simulated created maasmodelref.maas.opendatahub.io/e2e-distinct-simulated created maasmodelref.maas.opendatahub.io/e2e-embedding-simulated created maasmodelref.maas.opendatahub.io/e2e-trlp-test-simulated created maasmodelref.maas.opendatahub.io/e2e-unconfigured-facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/premium-simulated-simulated-premium created maassubscription.maas.opendatahub.io/premium-simulator-subscription created maassubscription.maas.opendatahub.io/simulator-subscription created llminferenceservice.serving.kserve.io/e2e-distinct-2-simulated created llminferenceservice.serving.kserve.io/e2e-distinct-simulated created llminferenceservice.serving.kserve.io/e2e-embedding-simulated created llminferenceservice.serving.kserve.io/e2e-trlp-test-simulated created llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium created ✅ MaaS system deployed (free + premium + e2e test fixtures) Waiting for models to be ready (timeout: 300s)... llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated condition met llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium condition met llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated condition met ✅ Simulator models ready Waiting for governed MaaSModelRefs to be Ready (timeout: 300s)... ✅ Governed MaaSModelRefs ready Waiting for MaaS Kuadrant AuthPolicies to be enforced (selector: app.kubernetes.io/managed-by=maas-controller, timeout: 180s)... ✅ All MaaS AuthPolicies enforced (1 policies) 2026-10-30T18:49:16Z deploy_models end Patching Authorino to log_level DEBUG... authorino.operator.authorino.kuadrant.io/authorino patched ✅ Authorino patched to log_level DEBUG deployment "authorino" successfully rolled out ---------------------------------------- Deploying ai-gateway-controller ---------------------------------------- 2026-10-30T18:49:16Z deploy_ai_gateway_controller start Deploying ai-gateway-controller image: quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-184 praxis-extproc image: quay.io/opendatahub/odh-praxis-extproc:odh-stable namespace: opendatahub gateway: openshift-ingress/maas-default-gateway customresourcedefinition.apiextensions.k8s.io/aiguardrails.aigateway.opendatahub.io created serviceaccount/ai-gateway-controller created clusterrole.rbac.authorization.k8s.io/ai-gateway-controller-role created clusterrolebinding.rbac.authorization.k8s.io/ai-gateway-controller-rolebinding created configmap/ai-gateway-controller-parameters created service/ai-gateway-controller-metrics created deployment.apps/ai-gateway-controller created servicemonitor.monitoring.coreos.com/ai-gateway-controller-metrics created networkpolicy.networking.k8s.io/ai-gateway-controller-allow-metrics-traffic created Waiting for deployment "ai-gateway-controller" rollout to finish: 0 of 1 updated replicas are available... deployment "ai-gateway-controller" successfully rolled out Triggering immediate praxis-extproc reconcile ... deployment.apps/ai-gateway-controller restarted Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... deployment "ai-gateway-controller" successfully rolled out Waiting for shared praxis-extproc (quay.io/opendatahub/odh-praxis-extproc:odh-stable) in openshift-ingress: payload-processing and payload-pre-processing (timeout: 300s) ... ✅ shared praxis-extproc ready: openshift-ingress/payload-processing and openshift-ingress/payload-pre-processing Annotating praxis IPP resources opendatahub.io/managed=false so maas-controller skips them ... deployment.apps/payload-processing annotated service/payload-processing annotated destinationrule.networking.istio.io/payload-processing annotated deployment.apps/payload-pre-processing annotated service/payload-pre-processing annotated destinationrule.networking.istio.io/payload-pre-processing annotated envoyfilter.networking.istio.io/payload-processing annotated networkpolicy.networking.k8s.io/payload-processing annotated Verified shared ext_proc dataplane image: openshift-ingress/payload-processing=quay.io/opendatahub/odh-praxis-extproc:odh-stable ai-gateway-controller rollout complete 2026-10-30T18:49:52Z deploy_ai_gateway_controller end ---------------------------------------- Enabling tenant namespace discovery ---------------------------------------- 2026-10-30T18:49:52Z tenant_namespace_discovery start Enabling --enable-tenant-namespace-discovery on maas-controller... maas-controller already has tenant namespace discovery enabled 2026-10-30T18:49:52Z tenant_namespace_discovery end ---------------------------------------- Setting up variables for tests ---------------------------------------- -- Setting up variables for tests -- HOST: maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev MAAS_API_BASE_URL: https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/maas-api ---------------------------------------- Setting up test tokens ---------------------------------------- Setting up premium test token (SA-based, works when oc whoami -t is unavailable)... Creating namespace: premium-users-namespace namespace/premium-users-namespace created Creating service account: premium-service-account serviceaccount/premium-service-account created Patching MaaSAuthPolicy premium-simulator-access to include system:serviceaccount:premium-users-namespace:premium-service-account... maasauthpolicy.maas.opendatahub.io/premium-simulator-access patched Patching MaaSSubscription premium-simulator-subscription to include system:serviceaccount:premium-users-namespace:premium-service-account... maassubscription.maas.opendatahub.io/premium-simulator-subscription patched Waiting for MaaSSubscriptions to reconcile after patch (timeout: 60s)... ✅ Both subscriptions ready: simulator-subscription=Active, premium-simulator-subscription=Active ✅ Premium test token setup complete (E2E_TEST_TOKEN_SA_* exported) Setting up test tokens (admin + regular user)... Current admin session: system:admin (will be preserved) ⚠️ No htpasswd token available - using SA token (admin tests may fail) Creating namespace: maas-admin namespace/maas-admin created Creating service account: tester-admin-user in maas-admin serviceaccount/tester-admin-user created Creating cluster role binding for tester-admin-user clusterrolebinding.rbac.authorization.k8s.io/tester-admin-user-binding created ✅ User setup completed: tester-admin-user (namespace: maas-admin) role.rbac.authorization.k8s.io/maas-admin-e2e created rolebinding.rbac.authorization.k8s.io/maas-admin-e2e-system-serviceaccount-maas-admin-tester-admin-user created auth.services.platform.opendatahub.io/auth patched ✅ Added system:serviceaccounts:maas-admin to Auth CR adminGroups (SA admin fallback) clusterrole.rbac.authorization.k8s.io/maas-admin created rolebinding.rbac.authorization.k8s.io/odh-admins-maas-admin created Creating separate SA token for regular user (required for IDOR tests)... Creating service account: tester-regular-user in default serviceaccount/tester-regular-user created Creating cluster role binding for tester-regular-user clusterrolebinding.rbac.authorization.k8s.io/tester-regular-user-binding created ✅ User setup completed: tester-regular-user (namespace: default) ✅ Regular user token for tester-regular-user (SA-based, namespace: default) Token setup complete (main session unchanged: system:admin) ---------------------------------------- Ensuring gateway allows model HTTPRoutes ---------------------------------------- /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 112: CUSTOM_RESOURCE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 113: NAMESPACE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 114: RESOURCE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 115: CRD_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 116: CSV_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 117: SUBSCRIPTION_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 118: POD_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 119: WEBHOOK_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 120: CUSTOM_CHECK_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 121: AUTHORINO_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 122: ROLLOUT_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 123: KUBECONFIG_WAIT_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 124: CATALOGSOURCE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 125: LLMIS_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 126: MAASMODELREF_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 127: AUTHPOLICY_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 130: _MAX_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 165: LOG_LEVEL_DEBUG: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 166: LOG_LEVEL_INFO: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 167: LOG_LEVEL_WARN: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 168: LOG_LEVEL_ERROR: readonly variable ./test/e2e/scripts/prow_run_ai_gateway_controller_test.sh: line 133: derive_infra_namespace: command not found Gateway openshift-ingress/maas-default-gateway already allows HTTPRoutes from llm ---------------------------------------- Validating Deployment ---------------------------------------- 2026-10-30T18:49:58Z validate start Deployment Validation ========================================= 🚀 MaaS Platform Deployment Validation ========================================= ========================================= 1️⃣ Component Status Checks ========================================= 🔍 Checking: MaaS API pods ✅ PASS: MaaS API has 1 running pod(s) 🔍 Checking: Policy engine pods (RHCL/Kuadrant) ✅ PASS: Policy engine has 7 running pod(s) in kuadrant-system 🔍 Checking: OpenDataHub/KServe pods ℹ️ opendatahub namespace: 12 running pod(s) ✅ PASS: OpenDataHub/RHOAI has 12 total running pod(s) 🔍 Checking: LLM namespace and models ✅ PASS: Found 7 LLMInferenceService(s) with 7 running pod(s) ========================================= 2️⃣ Gateway Status ========================================= 🔍 Checking: Gateway resource ✅ PASS: Gateway is Accepted and Programmed 🔍 Checking: HTTPRoute for maas-api ✅ PASS: HTTPRoute maas-api-route is configured and accepted 🔍 Checking: Gateway hostname ✅ PASS: Gateway hostname (from MAAS_GATEWAY_HOST): https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev ========================================= 3️⃣ Policy Status ========================================= 🔍 Checking: AuthPolicy ✅ PASS: AuthPolicy is configured and accepted 🔍 Checking: TokenRateLimitPolicy ✅ PASS: TokenRateLimitPolicy is configured and accepted ========================================= 4️⃣ API Endpoint Tests ========================================= ℹ️ Using gateway endpoint: https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev 🔍 Checking: Authentication token ✅ PASS: OpenShift identity token available 🔍 Checking: MaaS API key creation ✅ PASS: MaaS API key created (name: validate-test-1793386201) 🔍 Checking: Models endpoint ℹ️ Testing: curl -sSk https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/maas-api/v1/models -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" ✅ PASS: Models endpoint accessible, found 1 model(s) ℹ️ Available models: • publishers/llm/models/facebook/opt-125m - https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/ ℹ️ Using first available model: publishers/llm/models/facebook/opt-125m for validation 🔍 Checking: Model inference endpoint ℹ️ Testing: curl -sSk -X POST https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/v1/chat/completions -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"model": "publishers/llm/models/facebook/opt-125m", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50}' ✅ PASS: Model inference endpoint working ℹ️ Response: {"id":"chatcmpl-3670983a-10af-5d2b-b055-ede4dfdacb0b","created":1793386202,"model":"facebook/opt-125m","usage":{"prompt_tokens":6,"completion_tokens":25,"total_tokens":31},"object":"chat.completion"," 🔍 Checking: Rate limiting ℹ️ User tier: unknown (could not extract from token) ℹ️ Sending 10 rapid requests to test rate limiting... ✅ PASS: Rate limiting is working (5 successful, 5 rate limited) 🔍 Checking: Authorization enforcement (401 without token) ✅ PASS: Authorization is enforced (got 401 without token) ========================================= 📊 Validation Summary ========================================= Results: ✅ Passed: 15 ❌ Failed: 0 ⚠️ Warnings: 0 ✅ PASS: All critical checks passed! 🎉 Next steps: 1. Deploy a model: kustomize build docs/samples/models/simulator | kubectl apply -f - 2. Access the API at: https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev} 3. Check documentation: docs/README.md 4. Re-run validation with specific model: ./scripts/validate-deployment.sh MODEL_NAME Deployment validation completed 2026-10-30T18:50:05Z validate end ---------------------------------------- Running E2E Tests ---------------------------------------- -- E2E Tests (ai-gateway-controller, no external-model tests) -- Waiting for gateway: https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/maas-api/health ... Gateway reachable (HTTP 200) Waiting for authenticated gateway access ... Authenticated gateway access working (HTTP 200) Installing python3.11 (e2e requires Python >= 3.10; system python3 is too old)... Creating Python venv for e2e tests (/usr/bin/python3.11)... WARNING: The directory '/opt/app-root/src/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. WARNING: The directory '/opt/app-root/src/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. WARNING: The directory '/opt/app-root/src/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. Running E2E pass 1/2: parallel (E2E_PARALLEL_WORKERS=7, --dist=loadgroup, -m 'not serial') ============================= test session starts ============================== platform linux -- Python 3.11.13, pytest-9.2.0, pluggy-1.7.0 -- /workspace/source/test/e2e/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.142.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '9.2.0', 'pluggy': '1.7.0'}, 'Plugins': {'xdist': '3.8.0', 'metadata': '3.1.1', 'html': '4.2.0'}, 'PLATFORM': 'el9'} rootdir: /workspace/source/test/maas-e2e/test/e2e configfile: pyproject.toml plugins: xdist-3.8.0, metadata-3.1.1, html-4.2.0 created: 7/7 workers 7 workers [203 items] scheduling tests via LoadGroupScheduling test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_admin_manage_other_users_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_non_admin_cannot_access_other_users_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_list_api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_create_api_key test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_own_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_other_user_forbidden test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_revoke_api_key [gw1] [ 0%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_list_api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_by_subscription [gw1] [ 0%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_by_subscription test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[within-limit] [gw1] [ 1%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[within-limit] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[at-limit] [gw1] [ 1%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[at-limit] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[exceeds-limit] [gw1] [ 2%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[exceeds-limit] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[without-expiration] [gw1] [ 2%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[without-expiration] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[short-expiration] [gw1] [ 3%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[short-expiration] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_api_key_model_access_success [gw1] [ 3%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_api_key_model_access_success test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_invalid_api_key_rejected [gw1] [ 4%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_invalid_api_key_rejected test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_no_auth_header_rejected [gw1] [ 4%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_no_auth_header_rejected test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_revoked_api_key_rejected [gw1] [ 5%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_revoked_api_key_rejected test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_api_key_chat_completions [gw1] [ 5%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_api_key_chat_completions test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_double_revoke_returns_404 [gw1] [ 6%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_double_revoke_returns_404 test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_nonexistent_key_returns_404 [gw1] [ 6%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_nonexistent_key_returns_404 test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_then_create_new_key_works [gw1] [ 7%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_then_create_new_key_works test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_individual_revoke_multiple_keys [gw1] [ 7%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_individual_revoke_multiple_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_keys_rejected_at_gateway [gw1] [ 8%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_keys_rejected_at_gateway test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_cronjob_exists_and_configured [gw1] [ 8%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_cronjob_exists_and_configured test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_cleanup_networkpolicy_exists [gw1] [ 9%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_cleanup_networkpolicy_exists test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_create_ephemeral_key [gw1] [ 9%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_create_ephemeral_key test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_trigger_cleanup_preserves_active_keys [gw1] [ 10%] SKIPPED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_trigger_cleanup_preserves_active_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_active_subscription [gw2] [ 10%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_revoke_api_key test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_by_subscription_forbidden_for_non_admin [gw2] [ 11%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_by_subscription_forbidden_for_non_admin test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_failed_subscription [gw1] [ 11%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_active_subscription test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_degraded_subscription [gw2] [ 12%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_failed_subscription test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionFilter::test_search_filters_by_subscription [gw0] [ 12%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_create_api_key test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_admin_can_revoke_any_user [gw0] [ 13%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_admin_can_revoke_any_user test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_search_api_keys_by_labels [gw1] [ 13%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_degraded_subscription test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionFilter::test_search_without_subscription_returns_all [gw0] [ 14%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_search_api_keys_by_labels test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_labels_validation_errors [gw0] [ 14%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_labels_validation_errors test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_subscription_namespace_visible_to_api [gw0] [ 15%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_subscription_namespace_visible_to_api test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_another_namespace_not_visible_to_api [gw0] [ 15%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_another_namespace_not_visible_to_api test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_maas_subscription_namespace [gw0] [ 16%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_maas_subscription_namespace test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_another_namespace [gw0] [ 16%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_another_namespace test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_auth_policy_model_ref [gw0] [ 17%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_auth_policy_model_ref test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_subscription_model_ref [gw0] [ 17%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_subscription_model_ref test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAPIKeyManagementIsolation::test_api_key_cannot_mint_another_api_key [gw1] [ 18%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionFilter::test_search_without_subscription_returns_all test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_backward_compatibility_no_labels [gw1] [ 18%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_backward_compatibility_no_labels test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[group-only] [gw0] [ 19%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAPIKeyManagementIsolation::test_api_key_cannot_mint_another_api_key test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[username-only] [gw1] [ 19%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[group-only] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[keyname-only] [gw0] [ 20%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[username-only] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[identity-envelope] [gw1] [ 20%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[keyname-only] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[admin-scope] [gw0] [ 21%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[identity-envelope] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_injected_identity_headers_rejected_on_inference [gw0] [ 21%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_injected_identity_headers_rejected_on_inference test/maas-e2e/test/e2e/tests/test_negative_security.py::TestExpiredKeyRejection::test_expired_key_rejected_at_gateway [gw1] [ 22%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[admin-scope] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_duplicate_subscription_headers_ignored [gw0] [ 22%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestExpiredKeyRejection::test_expired_key_rejected_at_gateway test/maas-e2e/test/e2e/tests/test_negative_security.py::TestCrossModelAccess::test_key_cannot_access_model_outside_subscription [gw0] [ 23%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestCrossModelAccess::test_key_cannot_access_model_outside_subscription test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_authpolicy_with_nonexistent_model_ref [gw5] [ 23%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_own_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_combined_user_and_subscription [gw0] [ 24%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_authpolicy_with_nonexistent_model_ref test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderAbuse::test_special_characters_in_subscription_header [gw1] [ 24%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_duplicate_subscription_headers_ignored test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_subscription_with_nonexistent_model_ref [gw0] [ 25%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderAbuse::test_special_characters_in_subscription_header test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-cleanup] [gw0] [ 25%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-cleanup] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_health_endpoint_accessible [gw0] [ 26%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_health_endpoint_accessible test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_v1_models_via_maas_api_prefix [gw0] [ 26%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_v1_models_via_maas_api_prefix test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_authorized_user_gets_200 [gw0] [ 27%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_authorized_user_gets_200 test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_no_auth_gets_401 [gw0] [ 27%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_no_auth_gets_401 test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_invalid_token_gets_403 [gw0] [ 28%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_invalid_token_gets_403 test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_wrong_group_gets_403 [gw0] [ 28%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_wrong_group_gets_403 test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_uses_highest_priority_subscription [gw2] [ 29%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionFilter::test_search_filters_by_subscription test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_create_api_key_with_labels [gw2] [ 29%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_create_api_key_with_labels test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_nonexistent_subscription_errors [gw0] [ 30%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_uses_highest_priority_subscription test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_with_explicit_simulator_subscription [gw0] [ 30%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_with_explicit_simulator_subscription [gw2] [ 31%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_nonexistent_subscription_errors [gw6] [ 31%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_other_user_forbidden test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_missing_scope_returns_400 [gw6] [ 32%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_missing_scope_returns_400 test/maas-e2e/test/e2e/tests/test_subscription.py::TestManagedAnnotation::test_authpolicy_managed_false_prevents_update [gw6] [ 32%] SKIPPED test/maas-e2e/test/e2e/tests/test_subscription.py::TestManagedAnnotation::test_authpolicy_managed_false_prevents_update test/maas-e2e/test/e2e/tests/test_subscription.py::TestManagedAnnotation::test_trlp_managed_false_prevents_update [gw1] [ 33%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_subscription_with_nonexistent_model_ref test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-validate] [gw1] [ 33%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-validate] test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_subscription_but_no_access_gets_403 [gw5] [ 33%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_combined_user_and_subscription test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[subscriptions-select] [gw5] [ 34%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[subscriptions-select] test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_mint_api_key_denied_for_inaccessible_subscription test/maas-e2e/test/e2e/tests/test_subscription.py::TestOrderingEdgeCases::test_subscription_before_auth_policy test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_subscribed_user_gets_200 [gw2] [ 34%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_subscribed_user_gets_200 test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_active_status_with_valid_model [gw3] [ 35%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_admin_manage_other_users_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_dry_run [gw4] [ 35%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_non_admin_cannot_access_other_users_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_dry_run_by_subscription [gw3] [ 36%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_dry_run test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_active_status_with_valid_model [gw1] [ 36%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_subscription_but_no_access_gets_403 test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_multiple_subscriptions_separate_keys_gets_200 [gw0] [ 37%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestOrderingEdgeCases::test_subscription_before_auth_policy test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_subscription_but_no_auth_gets_403 [gw4] [ 37%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_dry_run_by_subscription [gw5] [ 38%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_mint_api_key_denied_for_inaccessible_subscription test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_access_gets_200 test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_failed_status_with_missing_model [gw0] [ 38%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_subscription_but_no_auth_gets_403 test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_status_transitions_on_model_deletion [gw6] [ 39%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestManagedAnnotation::test_trlp_managed_false_prevents_update test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_both_access_and_subscription_gets_200 [gw5] [ 39%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_access_gets_200 test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_degraded_healthy_model_allows_inference [gw1] [ 40%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_multiple_subscriptions_separate_keys_gets_200 test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_degraded_status_with_partial_models [gw5] [ 40%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_degraded_healthy_model_allows_inference test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_unauthenticated_returns_401[all-subscriptions] [gw2] [ 41%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_active_status_with_valid_model test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_failed_status_with_missing_model [gw2] [ 41%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_failed_status_with_missing_model test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_model_ref_display_name_and_description_enriched [gw2] [ 42%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_model_ref_display_name_and_description_enriched test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_returns_subscriptions_for_model [gw3] [ 42%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_active_status_with_valid_model test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_degraded_status_with_partial_models [gw2] [ 43%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_returns_subscriptions_for_model test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_unknown_model_returns_empty [gw2] [ 43%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_unknown_model_returns_empty test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestSubscriptionModelAccessFiltering::test_omits_subscription_with_no_authorized_models [gw4] [ 44%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_failed_status_with_missing_model test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_models_endpoint_with_degraded_subscription_api_key [gw6] [ 44%] FAILED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_both_access_and_subscription_gets_200 test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_returns_accessible_subscriptions [gw3] [ 45%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_degraded_status_with_partial_models test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestSubscriptionModelAccessFiltering::test_filters_unauthorized_models [gw4] [ 45%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_models_endpoint_with_degraded_subscription_api_key test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_deduplication_same_model_multiple_refs [gw2] [ 46%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestSubscriptionModelAccessFiltering::test_omits_subscription_with_no_authorized_models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_empty_subscription_header_value [gw2] [ 46%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_empty_subscription_header_value test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_response_schema_and_metadata [gw2] [ 47%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_response_schema_and_metadata test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_scoped_to_subscription [gw3] [ 47%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestSubscriptionModelAccessFiltering::test_filters_unauthorized_models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_user_token_returns_all_models [gw4] [ 48%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_deduplication_same_model_multiple_refs test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_user_token_with_subscription_header_filters [gw2] [ 48%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_scoped_to_subscription test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_with_deleted_subscription_403 [gw4] [ 49%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_user_token_with_subscription_header_filters test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_invalid_subscription_header_403 [gw0] [ 49%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_status_transitions_on_model_deletion test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_models_endpoint_with_degraded_subscription_kube_token [gw4] [ 50%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_invalid_subscription_header_403 test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_ignores_subscription_header [gw0] [ 50%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_models_endpoint_with_degraded_subscription_kube_token test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_api_keys_different_subscriptions [gw3] [ 51%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_user_token_returns_all_models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_with_inaccessible_subscription_403 [gw2] [ 51%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_with_deleted_subscription_403 test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_access_denied_to_subscription_403 [gw3] [ 52%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_with_inaccessible_subscription_403 test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_unauthenticated_request_401 [gw3] [ 52%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_unauthenticated_request_401 test/maas-e2e/test/e2e/tests/test_smoke.py::test_tokens_endpoint_replaced_by_api_keys [gw3] [ 53%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_tokens_endpoint_replaced_by_api_keys test/maas-e2e/test/e2e/tests/test_smoke.py::test_models_catalog [gw3] [ 53%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_models_catalog test/maas-e2e/test/e2e/tests/test_smoke.py::test_chat_completions_gateway_alive [gw3] [ 54%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_chat_completions_gateway_alive test/maas-e2e/test/e2e/tests/test_smoke.py::test_chat_completions_streaming_sse [gw3] [ 54%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_chat_completions_streaming_sse test/maas-e2e/test/e2e/tests/test_smoke.py::test_legacy_completions_optionally [gw3] [ 55%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_legacy_completions_optionally test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_tenant_ready_and_phase_healthy [gw3] [ 55%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_tenant_ready_and_phase_healthy test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_status_has_phase_and_conditions [gw3] [ 56%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_status_has_phase_and_conditions test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_spec_is_well_formed [gw3] [ 56%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_spec_is_well_formed test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_conditions_use_kubernetes_metav1_shape [gw3] [ 57%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_conditions_use_kubernetes_metav1_shape test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantNoFalseOwnership::test_maas_user_crs_not_owned_by_tenant [gw3] [ 57%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantNoFalseOwnership::test_maas_user_crs_not_owned_by_tenant test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_default_exists [gw3] [ 58%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_default_exists test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_not_terminating [gw3] [ 58%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_not_terminating test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_default_aitenant_lists_config_owner_reference [gw3] [ 59%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_default_aitenant_lists_config_owner_reference test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_tenant_config_lists_config_owner_reference [gw3] [ 59%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_tenant_config_lists_config_owner_reference test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_maas_controller_deployment_does_not_list_config_owner_reference [gw3] [ 60%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_maas_controller_deployment_does_not_list_config_owner_reference test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_requires_auth [gw5] [ 60%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_unauthenticated_returns_401[all-subscriptions] test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_subscription_includes_model_refs [gw4] [ 61%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_ignores_subscription_header test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_service_account_token_multiple_subs_no_header [gw3] [ 61%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_requires_auth test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_with_invalid_token [gw3] [ 62%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_with_invalid_token test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_not_exposed_through_gateway [gw3] [ 62%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_not_exposed_through_gateway test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_default_aitenant_bootstraps_maas_tenant_config_without_gateway_mutation [gw0] [ 63%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_api_keys_different_subscriptions test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_service_account_token_multiple_subs_with_header [gw2] [ 63%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_access_denied_to_subscription_403 test/maas-e2e/test/e2e/tests/test_smoke.py::test_healthz_or_404 [gw3] [ 64%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_default_aitenant_bootstraps_maas_tenant_config_without_gateway_mutation test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_rejected_outside_ai_tenants_namespace [gw2] [ 64%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_healthz_or_404 test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_migrates_and_removes_legacy_tenant [gw3] [ 65%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_rejected_outside_ai_tenants_namespace test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects [gw5] [ 65%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_subscription_includes_model_refs test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_authenticated [gw5] [ 66%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_authenticated test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_label_removal_stops_reconciliation [gw1] [ 66%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_degraded_status_with_partial_models test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_unauthenticated_returns_401[subscriptions-for-model] [gw1] [ 66%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_unauthenticated_returns_401[subscriptions-for-model] test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_dynamic_discovery_after_label_added [gw0] [ 67%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_service_account_token_multiple_subs_with_header test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_create_bootstrap_resources [gw4] [ 67%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_service_account_token_multiple_subs_no_header test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_gateway_matches_deployment [gw4] [ 68%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_gateway_matches_deployment test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_tenant_admin_rbac_is_namespace_scoped [gw6] [ 68%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_returns_accessible_subscriptions test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_distinct_models_in_subscription [gw5] [ 69%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_label_removal_stops_reconciliation test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_unlabeled_namespace_ignored [gw1] [ 69%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_dynamic_discovery_after_label_added test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_per_tenant_oidc_configuration [gw1] [ 70%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_per_tenant_oidc_configuration test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_same_tenant_access [gw6] [ 70%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_distinct_models_in_subscription test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantWebhookValidation::test_maasauthpolicy_rejected_without_tenant_config_cr [gw6] [ 71%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantWebhookValidation::test_maasauthpolicy_rejected_without_tenant_config_cr test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_unauthorized_access [gw2] [ 71%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_migrates_and_removes_legacy_tenant test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_derives_non_default_tenant_namespace [gw3] [ 72%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_labeled_tenant_namespace_is_discovered [gw0] [ 72%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_create_bootstrap_resources test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_namespace_qualified_collision_prevention [gw5] [ 73%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_unlabeled_namespace_ignored test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestLegacyDefaultNamespaceStillWorks::test_models_as_a_service_namespace_reconciles [gw5] [ 73%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestLegacyDefaultNamespaceStillWorks::test_models_as_a_service_namespace_reconciles test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_management_rules_have_model_identity_guards [gw5] [ 74%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_management_rules_have_model_identity_guards test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_gateway_default_auth_scoped_if_present [gw5] [ 74%] SKIPPED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_gateway_default_auth_scoped_if_present test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_full_tenant_lifecycle_create_to_delete [gw3] [ 75%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_labeled_tenant_namespace_is_discovered test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyLifecycle::test_gateway_auth_rego_is_fixed_size [gw3] [ 75%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyLifecycle::test_gateway_auth_rego_is_fixed_size test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_same_named_resources_across_tenants [gw4] [ 76%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_tenant_admin_rbac_is_namespace_scoped test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantWebhookValidation::test_maassubscription_rejected_without_tenant_config_cr [gw4] [ 76%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantWebhookValidation::test_maassubscription_rejected_without_tenant_config_cr test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_aitenant_creates_dedicated_maas_api_infrastructure [gw2] [ 77%] FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_derives_non_default_tenant_namespace test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyStructure::test_gateway_auth_policy_contract [gw2] [ 77%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyStructure::test_gateway_auth_policy_contract test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_service_routing_isolation [gw0] [ 78%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_namespace_qualified_collision_prevention test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyLifecycle::test_only_one_gateway_authpolicy_named_maas_gateway_auth [gw0] [ 78%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyLifecycle::test_only_one_gateway_authpolicy_named_maas_gateway_auth test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_default_and_multiple_tenants_coexist [gw1] [ 79%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_same_tenant_access test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_cross_tenant_isolation [gw1] [ 79%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_cross_tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_inference_succeeds_through_tenant_gateway[tenant_a] [gw6] [ 80%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_unauthorized_access test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_each_tenant_returns_own_gateway [gw6] [ 80%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_each_tenant_returns_own_gateway test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_auth_envoyfilters_select_their_gateway_workloads [gw3] [ 81%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_same_named_resources_across_tenants test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_tenant_namespace_label_change_triggers_reconciliation [gw3] [ 81%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_tenant_namespace_label_change_triggers_reconciliation test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_model_in_body_routing[correct-model] [gw0] [ 82%] ERROR test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_default_and_multiple_tenants_coexist test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_model_routes_through_tenant_gateway [gw1] [ 82%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_inference_succeeds_through_tenant_gateway[tenant_a] test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_inference_succeeds_through_tenant_gateway[tenant_b] [gw1] [ 83%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_inference_succeeds_through_tenant_gateway[tenant_b] test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_each_tenant_routes_to_own_model [gw1] [ 83%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_each_tenant_routes_to_own_model [gw6] [ 84%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_auth_envoyfilters_select_their_gateway_workloads test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_tenant_isolation_cross_gateway_blocked [gw6] [ 84%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_tenant_isolation_cross_gateway_blocked [gw3] [ 85%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_model_in_body_routing[correct-model] test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_model_in_body_routing[wrong-model] [gw3] [ 85%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_model_in_body_routing[wrong-model] [gw4] [ 86%] ERROR test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_aitenant_creates_dedicated_maas_api_infrastructure test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_tenant_name_environment_variable_set [gw4] [ 86%] ERROR test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_tenant_name_environment_variable_set [gw2] [ 87%] ERROR test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_service_routing_isolation test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_httproute_tenant_attachment [gw2] [ 87%] ERROR test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_httproute_tenant_attachment [gw5] [ 88%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_full_tenant_lifecycle_create_to_delete test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_default_tenant_unaffected_by_multitenancy_enablement [gw5] [ 88%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_default_tenant_unaffected_by_multitenancy_enablement [gw0] [ 89%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_model_routes_through_tenant_gateway test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_model_in_body_routing[missing-model] [gw0] [ 89%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_model_in_body_routing[missing-model] ==================================== ERRORS ==================================== _ ERROR at setup of TestPerTenantMaaSAPI.test_default_and_multiple_tenants_coexist _ [gw0] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_cases(): require_aitenant_crd() case_a = new_named_tenant_case("e2e-api-a") case_b = new_named_tenant_case("e2e-api-b") try: for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py:47: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1047: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-api-a-cfb433', namespace = 'ai-tenants' predicate = , timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-api-a-cfb433 in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-api-a-cfb433","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-api-a-cfb433"}}}\n'}, 'creationTimestamp': '2026-10-30T19:07:50Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-api-a-cfb433', 'namespace': 'ai-tenants', 'resourceVersion': '57351', 'uid': '69ec1a98-f5f9-4212-bec4-6472657cde4f'}, 'spec': {'gateway': {'name': 'e2e-api-a-cfb433'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:07:52Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-api-a-cfb433', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-api-a-cfb433'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:268: AssertionError _ ERROR at setup of TestTenantModelInference.test_inference_succeeds_through_tenant_gateway[tenant_a] _ [gw1] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1047: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-cc079d', namespace = 'ai-tenants' predicate = , timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-inf-a-cc079d in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-inf-a-cc079d","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-inf-a-cc079d"}}}\n'}, 'creationTimestamp': '2026-10-30T19:07:55Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-inf-a-cc079d', 'namespace': 'ai-tenants', 'resourceVersion': '57626', 'uid': 'bbcc4ea4-b6b4-43aa-a283-b3bb483ad40a'}, 'spec': {'gateway': {'name': 'e2e-inf-a-cc079d'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:07:58Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-inf-a-cc079d', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-inf-a-cc079d'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:268: AssertionError _ ERROR at setup of TestTenantModelInference.test_inference_succeeds_through_tenant_gateway[tenant_b] _ [gw1] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1047: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-cc079d', namespace = 'ai-tenants' predicate = , timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-inf-a-cc079d in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-inf-a-cc079d","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-inf-a-cc079d"}}}\n'}, 'creationTimestamp': '2026-10-30T19:07:55Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-inf-a-cc079d', 'namespace': 'ai-tenants', 'resourceVersion': '57626', 'uid': 'bbcc4ea4-b6b4-43aa-a283-b3bb483ad40a'}, 'spec': {'gateway': {'name': 'e2e-inf-a-cc079d'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:07:58Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-inf-a-cc079d', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-inf-a-cc079d'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:268: AssertionError _ ERROR at setup of TestTenantBodyRouting.test_each_tenant_routes_to_own_model _ [gw1] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1047: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-cc079d', namespace = 'ai-tenants' predicate = , timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-inf-a-cc079d in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-inf-a-cc079d","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-inf-a-cc079d"}}}\n'}, 'creationTimestamp': '2026-10-30T19:07:55Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-inf-a-cc079d', 'namespace': 'ai-tenants', 'resourceVersion': '57626', 'uid': 'bbcc4ea4-b6b4-43aa-a283-b3bb483ad40a'}, 'spec': {'gateway': {'name': 'e2e-inf-a-cc079d'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:07:58Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-inf-a-cc079d', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-inf-a-cc079d'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:268: AssertionError _ ERROR at setup of TestTenantModelInference.test_auth_envoyfilters_select_their_gateway_workloads _ [gw6] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1047: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-071cf0', namespace = 'ai-tenants' predicate = , timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-inf-a-071cf0 in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-inf-a-071cf0","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-inf-a-071cf0"}}}\n'}, 'creationTimestamp': '2026-10-30T19:08:40Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-inf-a-071cf0', 'namespace': 'ai-tenants', 'resourceVersion': '58762', 'uid': '9ba7e93c-eb92-435e-a582-0808d39ace13'}, 'spec': {'gateway': {'name': 'e2e-inf-a-071cf0'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:08:42Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-inf-a-071cf0', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-inf-a-071cf0'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:268: AssertionError _ ERROR at setup of TestTenantModelInference.test_tenant_isolation_cross_gateway_blocked _ [gw6] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1047: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-071cf0', namespace = 'ai-tenants' predicate = , timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-inf-a-071cf0 in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-inf-a-071cf0","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-inf-a-071cf0"}}}\n'}, 'creationTimestamp': '2026-10-30T19:08:40Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-inf-a-071cf0', 'namespace': 'ai-tenants', 'resourceVersion': '58762', 'uid': '9ba7e93c-eb92-435e-a582-0808d39ace13'}, 'spec': {'gateway': {'name': 'e2e-inf-a-071cf0'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:08:42Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-inf-a-071cf0', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-inf-a-071cf0'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:268: AssertionError _ ERROR at setup of TestTenantBodyRouting.test_model_in_body_routing[correct-model] _ [gw3] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1047: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-568107', namespace = 'ai-tenants' predicate = , timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-inf-a-568107 in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-inf-a-568107","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-inf-a-568107"}}}\n'}, 'creationTimestamp': '2026-10-30T19:11:14Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-inf-a-568107', 'namespace': 'ai-tenants', 'resourceVersion': '62758', 'uid': '4617c085-d2ef-4fb8-b220-d6394fdc716b'}, 'spec': {'gateway': {'name': 'e2e-inf-a-568107'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:11:16Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-inf-a-568107', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-inf-a-568107'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:268: AssertionError _ ERROR at setup of TestTenantBodyRouting.test_model_in_body_routing[wrong-model] _ [gw3] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:153: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1047: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-568107', namespace = 'ai-tenants' predicate = , timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-inf-a-568107 in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-inf-a-568107","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-inf-a-568107"}}}\n'}, 'creationTimestamp': '2026-10-30T19:11:14Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-inf-a-568107', 'namespace': 'ai-tenants', 'resourceVersion': '62758', 'uid': '4617c085-d2ef-4fb8-b220-d6394fdc716b'}, 'spec': {'gateway': {'name': 'e2e-inf-a-568107'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:11:16Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-inf-a-568107', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-inf-a-568107'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:268: AssertionError _ ERROR at setup of TestPerTenantMaaSAPI.test_aitenant_creates_dedicated_maas_api_infrastructure _ [gw4] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_cases(): require_aitenant_crd() case_a = new_named_tenant_case("e2e-api-a") case_b = new_named_tenant_case("e2e-api-b") try: for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py:47: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1047: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-api-b-8e31a7', namespace = 'ai-tenants' predicate = , timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-api-b-8e31a7 in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-api-b-8e31a7","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-api-b-8e31a7"}}}\n'}, 'creationTimestamp': '2026-10-30T19:09:43Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-api-b-8e31a7', 'namespace': 'ai-tenants', 'resourceVersion': '60479', 'uid': 'e2ed85a7-c201-4adb-bec8-860338ac0826'}, 'spec': {'gateway': {'name': 'e2e-api-b-8e31a7'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:09:44Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-api-b-8e31a7', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-api-b-8e31a7'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:268: AssertionError _ ERROR at setup of TestPerTenantMaaSAPI.test_tenant_name_environment_variable_set _ [gw4] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_cases(): require_aitenant_crd() case_a = new_named_tenant_case("e2e-api-a") case_b = new_named_tenant_case("e2e-api-b") try: for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py:47: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1047: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-api-b-8e31a7', namespace = 'ai-tenants' predicate = , timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-api-b-8e31a7 in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-api-b-8e31a7","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-api-b-8e31a7"}}}\n'}, 'creationTimestamp': '2026-10-30T19:09:43Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-api-b-8e31a7', 'namespace': 'ai-tenants', 'resourceVersion': '60479', 'uid': 'e2ed85a7-c201-4adb-bec8-860338ac0826'}, 'spec': {'gateway': {'name': 'e2e-api-b-8e31a7'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:09:44Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-api-b-8e31a7', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-api-b-8e31a7'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:268: AssertionError ____ ERROR at setup of TestPerTenantMaaSAPI.test_service_routing_isolation _____ [gw2] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_cases(): require_aitenant_crd() case_a = new_named_tenant_case("e2e-api-a") case_b = new_named_tenant_case("e2e-api-b") try: for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py:47: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1047: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-api-b-f3a0e7', namespace = 'ai-tenants' predicate = , timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-api-b-f3a0e7 in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-api-b-f3a0e7","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-api-b-f3a0e7"}}}\n'}, 'creationTimestamp': '2026-10-30T19:10:07Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-api-b-f3a0e7', 'namespace': 'ai-tenants', 'resourceVersion': '61221', 'uid': '92e80b4c-0451-44f8-9e7b-5bec881ecaed'}, 'spec': {'gateway': {'name': 'e2e-api-b-f3a0e7'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:10:09Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-api-b-f3a0e7', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-api-b-f3a0e7'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:268: AssertionError ___ ERROR at setup of TestPerTenantMaaSAPI.test_httproute_tenant_attachment ____ [gw2] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_cases(): require_aitenant_crd() case_a = new_named_tenant_case("e2e-api-a") case_b = new_named_tenant_case("e2e-api-b") try: for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py:47: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1047: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-api-b-f3a0e7', namespace = 'ai-tenants' predicate = , timeout = 180 interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: aitenant/e2e-api-b-f3a0e7 in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-api-b-f3a0e7","namespace":"ai-tenants"},"spec":{"gateway":{"name":"e2e-api-b-f3a0e7"}}}\n'}, 'creationTimestamp': '2026-10-30T19:10:07Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-api-b-f3a0e7', 'namespace': 'ai-tenants', 'resourceVersion': '61221', 'uid': '92e80b4c-0451-44f8-9e7b-5bec881ecaed'}, 'spec': {'gateway': {'name': 'e2e-api-b-f3a0e7'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:10:09Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-api-b-f3a0e7', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-api-b-f3a0e7'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:268: AssertionError =================================== FAILURES =================================== _ TestE2ESubscriptionFlow.test_e2e_with_both_access_and_subscription_gets_200 __ [gw6] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python self = def test_e2e_with_both_access_and_subscription_gets_200(self): """ API key with both access and subscription should get 200 OK. Uses an existing model reference that aligns with gateway path-based model identity. """ ns = _ns() auth_policy_name = "e2e-test-auth-success" subscription_name = "e2e-test-subscription-success" sa_name = "e2e-sa-success" try: # Create service account and get OC token for maas-api oc_token = _create_sa_token(sa_name, namespace=ns) sa_user = _sa_to_user(sa_name, namespace=ns) # Use pre-existing premium model to keep model identity consistent with gateway path. _create_test_auth_policy(auth_policy_name, PREMIUM_MODEL_REF, users=[sa_user]) _create_test_subscription(subscription_name, PREMIUM_MODEL_REF, users=[sa_user]) _wait_for_maas_auth_policy_phase(auth_policy_name, namespace=ns, timeout=120, require_auth_policies=False) > _wait_for_subscription_inference_ready( subscription_name, PREMIUM_MODEL_REF, namespace=ns, model_namespace=MODEL_NAMESPACE, timeout=180, ) test/maas-e2e/test/e2e/tests/test_subscription.py:1887: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_helper.py:1781: in _wait_for_subscription_inference_ready _wait_for_token_rate_limit_policy( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_ref = 'premium-w6-649d16' model_namespace = 'e2e-models-e2e-worker-w6-649d16', timeout = 180 subscription_name = 'e2e-test-subscription-success' subscription_namespace = 'ai-tenant-e2e-worker-w6-649d16' def _wait_for_token_rate_limit_policy( model_ref, model_namespace=MODEL_NAMESPACE, timeout=60, *, subscription_name=None, subscription_namespace=None ): """Wait for the current TokenRateLimitPolicy generation to be enforced. If a subscription is supplied, wait until the policy includes it. This prevents accepting the old policy before the controller updates its spec. Args: model_ref: Name of the model (e.g., "e2e-distinct-simulated") model_namespace: Namespace where the TRLP should be created (default: MODEL_NAMESPACE) timeout: Maximum wait time in seconds (default: 60) subscription_name: Subscription that the policy must include, if supplied subscription_namespace: Subscription namespace (defaults to _ns()) Raises: TimeoutError: If TRLP isn't created and enforced within timeout """ trlp_name = f"maas-trlp-{model_ref}" expected_subscription = ( f"{subscription_namespace or _ns()}/{subscription_name}" if subscription_name else None ) last_snapshot = "not found" deadline = time.time() + timeout log.info(f"Waiting for TokenRateLimitPolicy {trlp_name} in {model_namespace} (timeout: {timeout}s)...") while time.time() < deadline: result = subprocess.run( ["oc", "get", "tokenratelimitpolicy", trlp_name, "-n", model_namespace, "-o", "json"], capture_output=True, text=True, timeout=30, ) if result.returncode == 0: try: trlp = json.loads(result.stdout) metadata = trlp.get("metadata", {}) status = trlp.get("status", {}) generation = metadata.get("generation") observed = status.get("observedGeneration") conditions = status.get("conditions", []) enforced = next((c for c in conditions if c.get("type") == "Enforced"), None) subscriptions = { name.strip() for name in metadata.get("annotations", {}).get( "maas.opendatahub.io/subscriptions", "" ).split(",") } includes_subscription = expected_subscription is None or expected_subscription in subscriptions last_snapshot = ( f"generation={generation}, observedGeneration={observed}, " f"Enforced={enforced}, includes_subscription={includes_subscription}" ) if ( enforced and enforced.get("status") == "True" and generation is not None and observed == generation and includes_subscription ): log.info(f"TokenRateLimitPolicy {trlp_name} generation {generation} is enforced") return log.debug(f"TokenRateLimitPolicy {trlp_name} is not ready: {last_snapshot}") except (json.JSONDecodeError, KeyError) as e: log.debug(f"Failed to parse TRLP status: {e}") elif _is_not_found_error(result.stderr): log.debug(f"TokenRateLimitPolicy {trlp_name} not found yet...") elif _is_transient_kubectl_error(result.stderr): log.debug( f"Transient error while reading TokenRateLimitPolicy {trlp_name}: {result.stderr.strip()}" ) else: raise RuntimeError( f"Failed to get TokenRateLimitPolicy {trlp_name} in namespace '{model_namespace}': " f"{result.stderr.strip()}" ) time.sleep(3) > raise TimeoutError( f"TokenRateLimitPolicy {trlp_name} was not created and enforced in {model_namespace} within {timeout}s: {last_snapshot}" ) E TimeoutError: TokenRateLimitPolicy maas-trlp-premium-w6-649d16 was not created and enforced in e2e-models-e2e-worker-w6-649d16 within 180s: generation=2, observedGeneration=2, Enforced={'lastTransitionTime': '2026-10-30T18:55:58Z', 'message': 'TokenRateLimitPolicy waiting for the following components to sync: [Limitador]', 'reason': 'Unknown', 'status': 'False', 'type': 'Enforced'}, includes_subscription=True test/maas-e2e/test/e2e/tests/test_helper.py:1538: TimeoutError ------------------------------ Captured log setup ------------------------------ INFO test_subscription:test_subscription.py:1819 ============================================================ INFO test_subscription:test_subscription.py:1820 Validating E2E Test Prerequisites INFO test_subscription:test_subscription.py:1821 ============================================================ INFO test_subscription:test_subscription.py:1836 ✓ Model 'simulator-w6-649d16' is Ready INFO test_subscription:test_subscription.py:1837 Endpoint: https://e2e-worker-w6-649d16.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/ INFO test_subscription:test_subscription.py:1843 ✓ Found 1 auth policies for model: INFO test_subscription:test_subscription.py:1845 - simulator-access INFO test_subscription:test_subscription.py:1847 ✓ Found 1 subscriptions for model: INFO test_subscription:test_subscription.py:1849 - simulator-subscription INFO test_subscription:test_subscription.py:1862 ============================================================ INFO test_subscription:test_subscription.py:1863 ✅ All prerequisites validated - proceeding with tests INFO test_subscription:test_subscription.py:1864 ============================================================ ------------------------------ Captured log call ------------------------------- INFO test_helper:test_helper.py:1070 Creating MaaSAuthPolicy: e2e-test-auth-success INFO test_helper:test_helper.py:1135 Creating MaaSSubscription: e2e-test-subscription-success INFO test_helper:test_helper.py:1821 Waiting for MaaSAuthPolicy e2e-test-auth-success to reach phase 'Active' (timeout: 120s)... INFO test_helper:test_helper.py:1399 Waiting for gateway AuthPolicy openshift-ingress/e2e-worker-w6-649d16-maas-auth Accepted+Enforced (timeout: 240s)... INFO test_helper:test_helper.py:1448 Gateway AuthPolicy openshift-ingress/e2e-worker-w6-649d16-maas-auth is Accepted and Enforced INFO test_helper:test_helper.py:1846 MaaSAuthPolicy e2e-test-auth-success reached phase 'Active' INFO test_helper:test_helper.py:1633 MaaSSubscription ai-tenant-e2e-worker-w6-649d16/e2e-test-subscription-success is discovery-ready in phase Active INFO test_helper:test_helper.py:1486 Waiting for TokenRateLimitPolicy maas-trlp-premium-w6-649d16 in e2e-models-e2e-worker-w6-649d16 (timeout: 180s)... ___ TestAITenantLifecycle.test_aitenant_derives_non_default_tenant_namespace ___ [gw2] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python self = def test_aitenant_derives_non_default_tenant_namespace(self): """RHOAIENG-66836: non-default AITenant must not use models-as-a-service tenant namespace.""" suffix = uuid.uuid4().hex[:8] aitenant_name = f"e2e-derive-{suffix}" reserved_ns = _ns() expected_ns = f"ai-tenant-{aitenant_name}" gateway_name = aitenant_name try: _apply_gateway_fixture({"gateway_name": gateway_name, "aitenant_name": aitenant_name}) _apply( { "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "AITenant", "metadata": {"name": aitenant_name, "namespace": AITENANT_NAMESPACE}, "spec": {}, } ) > aitenant = _wait_for_json( AITENANT_KIND, aitenant_name, AITENANT_NAMESPACE, predicate=_aitenant_ready, timeout=120, ) test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:610: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-derive-4cef683f', namespace = 'ai-tenants' predicate = , timeout = 120 interval = 5 def _wait_for_json(kind, name, namespace=None, *, predicate=None, timeout=180, interval=5): deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = _get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError(f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}") E AssertionError: aitenant/e2e-derive-4cef683f in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"AITenant","metadata":{"annotations":{},"name":"e2e-derive-4cef683f","namespace":"ai-tenants"},"spec":{}}\n'}, 'creationTimestamp': '2026-10-30T19:03:49Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup'], 'generation': 1, 'name': 'e2e-derive-4cef683f', 'namespace': 'ai-tenants', 'resourceVersion': '49604', 'uid': 'b0fd81af-0a66-4f8b-ab26-71381438dc6c'}, 'spec': {}, 'status': {'conditions': [{'lastTransitionTime': '2026-10-30T19:03:51Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'e2e-derive-4cef683f', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'ai-tenant-e2e-derive-4cef683f'}} test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:113: AssertionError ------ generated xml file: /workspace/artifacts-dir/e2e-system:admin.xml ------- - Generated html report: file:///workspace/artifacts-dir/e2e-system%3Aadmin.html - =========================== short test summary info ============================ FAILED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_both_access_and_subscription_gets_200 FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_derives_non_default_tenant_namespace ERROR test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_default_and_multiple_tenants_coexist ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_inference_succeeds_through_tenant_gateway[tenant_a] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_inference_succeeds_through_tenant_gateway[tenant_b] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_each_tenant_routes_to_own_model ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_auth_envoyfilters_select_their_gateway_workloads ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_tenant_isolation_cross_gateway_blocked ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_model_in_body_routing[correct-model] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_model_in_body_routing[wrong-model] ERROR test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_aitenant_creates_dedicated_maas_api_infrastructure ERROR test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_tenant_name_environment_variable_set ERROR test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_service_routing_isolation ERROR test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_httproute_tenant_attachment !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 14 failures !!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! = 2 failed, 153 passed, 15 skipped, 128 warnings, 12 errors in 1876.88s (0:31:16) = Running E2E pass 2/2: serial cluster mutators (-m serial, single worker) ============================= test session starts ============================== platform linux -- Python 3.11.13, pytest-9.2.0, pluggy-1.7.0 -- /workspace/source/test/e2e/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.142.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '9.2.0', 'pluggy': '1.7.0'}, 'Plugins': {'xdist': '3.8.0', 'metadata': '3.1.1', 'html': '4.2.0'}, 'PLATFORM': 'el9'} rootdir: /workspace/source/test/maas-e2e/test/e2e configfile: pyproject.toml plugins: xdist-3.8.0, metadata-3.1.1, html-4.2.0 collecting ... collected 240 items / 203 deselected / 37 selected test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_pending_subscription [token] using env TOKEN (masked): 1037 PASSED [ 2%] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_reject_key_for_unreconciled_subscription PASSED [ 5%] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAuthPolicyRemoval::test_authpolicy_deletion_revokes_access PASSED [ 8%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_auth_pass_no_subscription_gets_403 PASSED [ 10%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_rate_limit_exhaustion_gets_429 PASSED [ 13%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_models_endpoint_exempt_from_rate_limiting PASSED [ 16%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestMultipleSubscriptionsPerModel::test_user_in_one_of_two_subscriptions_gets_200 PASSED [ 18%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestUnlimitedSubscription::test_unlimited_and_token_rate_limits_are_mutually_exclusive PASSED [ 21%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestUnlimitedSubscription::test_unlimited_key_is_not_rate_limited PASSED [ 24%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestUnlimitedSubscription::test_unlimited_subscriptions_share_one_wasm_limit SKIPPED [ 27%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestUnlimitedSubscription::test_unlimited_usage_is_metered PASSED [ 29%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestAllUnlimitedModel::test_all_unlimited_model_is_not_denied PASSED [ 32%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestMultipleAuthPoliciesPerModel::test_two_auth_policies_or_logic PASSED [ 35%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestMultipleAuthPoliciesPerModel::test_delete_one_auth_policy_other_still_works PASSED [ 37%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestCascadeDeletion::test_delete_subscription_rebuilds_trlp PASSED [ 40%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestCascadeDeletion::test_trlp_persists_during_multi_subscription_deletion PASSED [ 43%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestCascadeDeletion::test_delete_last_subscription_denies_access PASSED [ 45%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestCascadeDeletion::test_unconfigured_model_denied_by_gateway_auth PASSED [ 48%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_access_but_no_subscription_gets_403 PASSED [ 51%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_single_subscription_auto_selects PASSED [ 54%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_auth_but_no_subscription_gets_403 PASSED [ 56%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_degraded_trlp_blocks_inference PASSED [ 59%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_failed_subscription_blocks_inference PASSED [ 62%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_single_subscription_auto_select PASSED [ 64%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_explicit_subscription_header maassubscription.maas.opendatahub.io/premium-simulator-subscription patched maassubscription.maas.opendatahub.io/premium-simulator-subscription patched PASSED [ 67%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_models_filtered_by_subscription maassubscription.maas.opendatahub.io/premium-simulator-subscription patched maassubscription.maas.opendatahub.io/premium-simulator-subscription patched PASSED [ 70%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_different_modelrefs_same_model_id maasauthpolicy.maas.opendatahub.io/e2e-diff-refs-auth created maassubscription.maas.opendatahub.io/e2e-diff-refs-subscription created PASSED [ 72%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_empty_model_list PASSED [ 75%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_central_models_endpoint_exempt_from_rate_limiting PASSED [ 78%] test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_payload_processing_deployed_with_active_tenant PASSED [ 81%] test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_payload_processing_and_pre_processing_resources SKIPPED [ 83%] test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantDiscoveryDormantMode::test_dormant_mode_ignores_labeled_namespace SKIPPED [ 86%] test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingDefaultDeny::test_embedding_default_deny_403 PASSED [ 89%] test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingPathRouting::test_embedding_path_based_200 PASSED [ 91%] test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingPathRouting::test_embedding_bbr_llmisvc_200 PASSED [ 94%] test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingGovernance::test_embedding_trlp_429 PASSED [ 97%] test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingGovernance::test_embedding_with_governance_200 PASSED [100%] --- generated xml file: /workspace/artifacts-dir/e2e-system:admin-serial.xml --- - Generated html report: file:///workspace/artifacts-dir/e2e-system%3Aadmin-serial.html - ==== 34 passed, 3 skipped, 203 deselected, 45 warnings in 962.92s (0:16:02) ==== ERROR: E2E tests failed (parallel_rc=1, serial_rc=0) ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Saved to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs Saved 9 resource type(s) to /workspace/artifacts-dir/maas-crs Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 12 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Collecting pod logs from namespace odh-ai-gateway-infra to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Saved 10 pod log file(s) to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Collecting pod logs from namespace models-as-a-service to /workspace/artifacts-dir/pod-logs/models-as-a-service Saved 0 0 pod log file(s) to /workspace/artifacts-dir/pod-logs/models-as-a-service Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Collecting pod logs from namespace kuadrant-system to /workspace/artifacts-dir/pod-logs/kuadrant-system Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/kuadrant-system Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 17 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Collecting pod logs from namespace llm to /workspace/artifacts-dir/pod-logs/llm Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/llm Skipping namespace istio-system (not found) ============================================== ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Saved to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs Saved 10 resource type(s) to /workspace/artifacts-dir/maas-crs Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 12 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Collecting pod logs from namespace odh-ai-gateway-infra to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Saved 10 pod log file(s) to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Collecting pod logs from namespace models-as-a-service to /workspace/artifacts-dir/pod-logs/models-as-a-service Saved 0 0 pod log file(s) to /workspace/artifacts-dir/pod-logs/models-as-a-service Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Collecting pod logs from namespace kuadrant-system to /workspace/artifacts-dir/pod-logs/kuadrant-system Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/kuadrant-system Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 17 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Collecting pod logs from namespace llm to /workspace/artifacts-dir/pod-logs/llm Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/llm Skipping namespace istio-system (not found) ============================================== Collecting AIGC cluster CR extensions to /workspace/artifacts-dir/maas-crs Saved datascienceclusters.yaml Saved httproutes.yaml Saved gateways.yaml Collecting MaaS diagnostics to /workspace/artifacts-dir/gather-maas (log: /workspace/artifacts-dir/maas-must-gather.log) ... MaaS must-gather complete (see /workspace/artifacts-dir/maas-must-gather.log) ======================================== Cluster / Namespace Info ======================================== --- Current context --- admin --- Logged-in user --- system:admin --- Cluster domain --- apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev DEPLOYMENT_NAMESPACE: opendatahub MAAS_API_DEPLOYMENT_NAMESPACE: odh-ai-gateway-infra MAAS_SUBSCRIPTION_NAMESPACE: models-as-a-service AUTHORINO_NAMESPACE: kuadrant-system ======================================== MaaS API Deployment ======================================== --- maas-api pods --- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES maas-api-595cc648c-b9fk4 1/1 Running 0 2m45s 10.133.0.137 ip-10-0-140-122.us-west-2.compute.internal maas-api-595cc648c-dzb2m 1/1 Running 0 2m37s 10.134.0.127 ip-10-0-128-92.us-west-2.compute.internal maas-api-revoke-sub-0165d7d2-fxkl8 0/1 Completed 0 115s 10.132.1.17 ip-10-0-131-224.us-west-2.compute.internal maas-api-revoke-sub-3e31a506-gpbc5 0/1 Completed 0 3m29s 10.132.1.14 ip-10-0-131-224.us-west-2.compute.internal maas-api-revoke-sub-452b6d7b-gf6xw 0/1 Completed 0 3m55s 10.132.1.13 ip-10-0-131-224.us-west-2.compute.internal maas-api-revoke-sub-6b19338f-h8v98 0/1 Completed 0 104s 10.132.1.18 ip-10-0-131-224.us-west-2.compute.internal maas-api-revoke-sub-6d2a8c49-ht8t2 0/1 Completed 0 2m59s 10.132.1.15 ip-10-0-131-224.us-west-2.compute.internal maas-api-revoke-sub-f03629db-z9x5m 0/1 Completed 0 2m16s 10.132.1.16 ip-10-0-131-224.us-west-2.compute.internal --- maas-api service --- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR maas-api ClusterIP 172.31.197.207 8443/TCP 51m app.kubernetes.io/component=api,app.kubernetes.io/name=maas-api,app.kubernetes.io/part-of=models-as-a-service,maas.opendatahub.io/tenant-instance=maas-api ======================================== maas-controller ======================================== --- maas-controller pods --- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES maas-controller-7496565bcf-r9dth 1/1 Running 0 16m 10.133.0.128 ip-10-0-140-122.us-west-2.compute.internal --- maas-controller namespace env --- INFRA_NAMESPACE=AUTO ======================================== Kuadrant Policies ======================================== --- AuthPolicies (all namespaces) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE openshift-ingress maas-gateway-auth True True Gateway maas-default-gateway 52m --- TokenRateLimitPolicies (all namespaces) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE llm maas-trlp-facebook-opt-125m-simulated True True HTTPRoute facebook-opt-125m-simulated-kserve-route 8m57s llm maas-trlp-premium-simulated-simulated-premium True True HTTPRoute premium-simulated-simulated-premium-kserve-route 50m openshift-ingress gateway-default-deny True False Gateway maas-default-gateway 51m ======================================== MaaS CRs (maas.opendatahub.io) ======================================== --- Configs (cluster-scoped) --- NAME READY AGE default True 52m --- AITenants (all namespaces) --- NAMESPACE NAME READY TENANT NAMESPACE GATEWAY AGE ai-tenants models-as-a-service True models-as-a-service maas-default-gateway 51m --- Tenants (all namespaces) --- --- MaaSAuthPolicies --- NAME PHASE AGE AUTHPOLICIES premium-simulator-access Active 51m simulator-access Active 51m --- MaaSSubscriptions --- NAME PHASE PRIORITY AGE premium-simulator-subscription Active 20 51m simulator-subscription Active 10 8m59s --- MaaSSubscription status details --- premium-simulator-subscription: Active - successfully reconciled simulator-subscription: Active - successfully reconciled --- MaaSModelRefs (all namespaces) --- NAMESPACE NAME PHASE ENDPOINT HTTPROUTE GATEWAY AGE e2e-models-e2e-worker-single-main-34d64c simulator-single-main-34d64c Failed simulator-single-main-34d64c-kserve-route e2e-worker-single-main-34d64c 5m37s e2e-models-e2e-worker-single-main-7d30a0 simulator-single-main-7d30a0 Failed simulator-single-main-7d30a0-kserve-route e2e-worker-single-main-7d30a0 10m e2e-models-e2e-worker-w0-f5705f distinct-2-w0-f5705f Failed distinct-2-w0-f5705f-kserve-route e2e-worker-w0-f5705f 39m e2e-models-e2e-worker-w0-f5705f distinct-w0-f5705f Failed distinct-w0-f5705f-kserve-route e2e-worker-w0-f5705f 41m e2e-models-e2e-worker-w0-f5705f premium-w0-f5705f Failed premium-w0-f5705f-kserve-route e2e-worker-w0-f5705f 45m e2e-models-e2e-worker-w0-f5705f simulator-w0-f5705f Failed simulator-w0-f5705f-kserve-route e2e-worker-w0-f5705f 46m e2e-models-e2e-worker-w5-3b6ad6 distinct-2-w5-3b6ad6 Pending distinct-2-w5-3b6ad6-kserve-route e2e-worker-w5-3b6ad6 38m e2e-models-e2e-worker-w5-3b6ad6 distinct-w5-3b6ad6 Pending distinct-w5-3b6ad6-kserve-route e2e-worker-w5-3b6ad6 40m e2e-models-e2e-worker-w5-3b6ad6 premium-w5-3b6ad6 Pending premium-w5-3b6ad6-kserve-route e2e-worker-w5-3b6ad6 45m e2e-models-e2e-worker-w5-3b6ad6 simulator-w5-3b6ad6 Pending simulator-w5-3b6ad6-kserve-route e2e-worker-w5-3b6ad6 46m llm e2e-distinct-2-simulated Pending e2e-distinct-2-simulated-kserve-route maas-default-gateway 51m llm e2e-distinct-simulated Pending e2e-distinct-simulated-kserve-route maas-default-gateway 51m llm e2e-embedding-simulated Pending e2e-embedding-simulated-kserve-route maas-default-gateway 51m llm e2e-trlp-test-simulated Pending e2e-trlp-test-simulated-kserve-route maas-default-gateway 51m llm e2e-unconfigured-facebook-opt-125m-simulated Pending e2e-unconfigured-facebook-opt-125m-simulated-kserve-route maas-default-gateway 51m llm facebook-opt-125m-simulated Ready https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/ facebook-opt-125m-simulated-kserve-route maas-default-gateway 51m llm premium-simulated-simulated-premium Ready https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/ premium-simulated-simulated-premium-kserve-route maas-default-gateway 51m --- MaasTenantConfigs --- NAME READY REASON AGE INFRANAMESPACE default-tenant True Reconciled 51m odh-ai-gateway-infra --- MaasTenantConfig status details --- default-tenant: True - MaaS platform manifests applied and maas-api deployment is available --- ExternalModels (maas group) --- ======================================== Inference CRs (inference.opendatahub.io) ======================================== --- ExternalModels (inference group) --- --- ExternalProviders (inference group) --- ======================================== AI Gateway CRs ======================================== --- AIGateways (cluster-scoped) --- --- LLMBatchGateways (all namespaces) --- ======================================== KServe Resources ======================================== --- LLMInferenceServices (all namespaces) --- NAMESPACE NAME URL READY REASON AGE URLS llm e2e-distinct-2-simulated https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/llm/e2e-distinct-2-simulated True 51m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-distinct-simulated https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/llm/e2e-distinct-simulated True 51m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-embedding-simulated https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/llm/e2e-embedding-simulated True 51m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-trlp-test-simulated https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/llm/e2e-trlp-test-simulated True 51m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-unconfigured-facebook-opt-125m-simulated https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/llm/e2e-unconfigured-facebook-opt-125m-simulated True 51m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm facebook-opt-125m-simulated https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/llm/facebook-opt-125m-simulated True 50m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm premium-simulated-simulated-premium https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/llm/premium-simulated-simulated-premium True 50m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ ======================================== Kuadrant Policies (managed by maas-controller) ======================================== --- AuthPolicies (managed) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE openshift-ingress maas-gateway-auth True True Gateway maas-default-gateway 52m --- TokenRateLimitPolicies (managed) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE llm maas-trlp-facebook-opt-125m-simulated True True HTTPRoute facebook-opt-125m-simulated-kserve-route 9m1s llm maas-trlp-premium-simulated-simulated-premium True True HTTPRoute premium-simulated-simulated-premium-kserve-route 50m openshift-ingress gateway-default-deny True False Gateway maas-default-gateway 51m ======================================== Payload Processing / IPP ======================================== --- IPP Deployments (openshift-ingress) --- --- IPP Services (openshift-ingress) --- --- EnvoyFilters (openshift-ingress) --- NAME AGE data-science-authn-filter 57m kuadrant-auth-maas-default-gateway 51m kuadrant-maas-default-gateway 51m kuadrant-ratelimiting-maas-default-gateway 51m maas-default-gateway-authn-ssl 53m maas-model-access-logs 51m payload-processing 49m --- IPP ConfigMaps (openshift-ingress) --- ======================================== NetworkPolicies ======================================== --- NetworkPolicies (openshift-ingress) --- NAME POD-SELECTOR AGE data-science-gateway-allow gateway.networking.k8s.io/gateway-name=data-science-gateway 57m istiod-allow app=istiod 57m kube-auth-proxy app=kube-auth-proxy 57m maas-default-gateway-allow gateway.networking.k8s.io/gateway-name=maas-default-gateway 53m openshift-ingress-deny-all 69m payload-processing maas.opendatahub.io/tenant-instance in (payload-pre-processing,payload-processing) 49m router-default ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default 69m ======================================== Test User Information ======================================== No test token available (not logged in via oc) ======================================== Subscription → Model Mapping ======================================== Subscription: premium-simulator-subscription Owner users: (none) Owner groups: premium-user Models: llm/premium-simulated-simulated-premium Subscription: simulator-subscription Owner users: (none) Owner groups: system:authenticated Models: llm/facebook-opt-125m-simulated ======================================== Available Models (MaaSModelRefs) ======================================== Model Reference → Model ID / Endpoint e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c → simulator-single-main-34d64c (Failed) e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0 → simulator-single-main-7d30a0 (Failed) e2e-models-e2e-worker-w0-f5705f/distinct-2-w0-f5705f → distinct-2-w0-f5705f (Failed) e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f → distinct-w0-f5705f (Failed) e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f → premium-w0-f5705f (Failed) e2e-models-e2e-worker-w0-f5705f/simulator-w0-f5705f → simulator-w0-f5705f (Failed) e2e-models-e2e-worker-w5-3b6ad6/distinct-2-w5-3b6ad6 → distinct-2-w5-3b6ad6 (Pending) e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6 → distinct-w5-3b6ad6 (Pending) e2e-models-e2e-worker-w5-3b6ad6/premium-w5-3b6ad6 → premium-w5-3b6ad6 (Pending) e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6 → simulator-w5-3b6ad6 (Pending) llm/e2e-distinct-2-simulated → e2e-distinct-2-simulated (Pending) llm/e2e-distinct-simulated → e2e-distinct-simulated (Pending) llm/e2e-embedding-simulated → e2e-embedding-simulated (Pending) llm/e2e-trlp-test-simulated → e2e-trlp-test-simulated (Pending) llm/e2e-unconfigured-facebook-opt-125m-simulated → e2e-unconfigured-facebook-opt-125m-simulated (Pending) llm/facebook-opt-125m-simulated → facebook-opt-125m-simulated (Ready) Endpoint: https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/ llm/premium-simulated-simulated-premium → premium-simulated-simulated-premium (Ready) Endpoint: https://maas.apps.65b65f5330ae8ee2f020.prod.konflux-ocp-ci.dev/ ======================================== Gateway / HTTPRoutes ======================================== --- Gateway --- NAME CLASS ADDRESS PROGRAMMED AGE maas-default-gateway openshift-default maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local True 53m --- HTTPRoutes (maas-api) --- NAME HOSTNAMES AGE maas-api-route 51m ======================================== Authorino ======================================== --- Authorino pods --- --- ======================================== Subscription Selector Endpoint Validation ======================================== Expected URL (from resolved maas-api deployment namespace): https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select (MAAS_API_DEPLOYMENT_NAMESPACE resolved to: odh-ai-gateway-infra) --- Sample AuthPolicy subscription-info configuration --- Inspecting: openshift-ingress/maas-gateway-auth Actual URL in AuthPolicy: https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select ✅ Request body includes requestedModel field Model reference: / Request body preview: { "groups": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.groups : (has(auth.identity.iss) ? (has(auth.identity.groups) && auth.identity.groups.all(g, g.matches('^[A-Za-z0-9:#._/ -]+$')) ? auth.identity.groups : []) : (has(auth.identity.user.groups) && auth.identity.user.groups.all(g, g.matches('^[A-Za-z0-9:#._/ -]+$')) ? auth.identity.user.groups : [])), "username": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.username : (has(auth.identity.preferred_username) ? auth.identity.preferred_username : (has(auth.identity.sub) ? auth.identity.sub : auth.identity.user.username)), "requestedSubscription": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.subscription : ("x-maas-subscription" in request.headers ? request.headers["x-maas-subscription"] : ""), "requestedModel": (size(request.path.split("/").filter(x, x != "")) >= 2 && request.path.split("/").filter(x, x != "")[0] != "v1" && request.path.split("/").filter(x, x != "")[0] != "maas-api" ? request.path.split("/").filter(x, x != "")[0] + "/" + request.path.split("/").filter(x, x != "")[1] : ("x-gateway-model-name" in request.headers ? request.headers["x-gateway-model-name"] : "")) --- Connectivity test (from kuadrant-system, simulates Authorino) --- curl -vsk -m 10 -X POST 'https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select' -H 'Content-Type: application/json' -d '{}' kubectl run failed or timed out ======================================== DNS Resolution Check ======================================== Resolving: maas-api.odh-ai-gateway-infra.svc.cluster.local nslookup failed ======================================== Configuration Summary ======================================== This summary helps compare local vs CI runs: AITenants (all namespaces): 1 MaaSModelRefs (all namespaces): 17 MaaSSubscriptions (models-as-a-service): 2 MaaSAuthPolicies (models-as-a-service): 2 Generated Kuadrant AuthPolicies: 1 Generated Kuadrant TokenRateLimitPolicies: 3 LLMInferenceServices (all namespaces): 7 ExternalModels - inference (all namespaces): 0 ExternalProviders - inference (all namespaces): 0 EnvoyFilters (openshift-ingress): 7 IPP Deployments (openshift-ingress): 0 NetworkPolicies (openshift-ingress): 7 Subscription selector URL: https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select Test user: system:admin