Fetching https://github.com/jland-redhat/models-as-a-service@630e7b50303688a6e3f25cd039e32822f38d6e28 into /workspace/source/test/maas-e2e ... MaaS checkout at: 630e7b50303688a6e3f25cd039e32822f38d6e28 Patched /workspace/source/test/maas-e2e/scripts/deploy.sh Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS checkout fixture root) Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS AuthPolicy wait scope) MaaS platform images: MAAS_API_IMAGE=quay.io/opendatahub/maas-api:latest MAAS_CONTROLLER_IMAGE=quay.io/opendatahub/maas-controller:latest AI_GATEWAY_CONTROLLER_IMAGE=quay.io/opendatahub/odh-ai-gateway-controller@sha256:fd19843e47bfe605ec44d877133464e5b56e3298fae56226e9eaaae0f07c1dc6 PRAXIS_EXTPROC_IMAGE=quay.io/opendatahub/odh-praxis-extproc:odh-stable ---------------------------------------- ai-gateway-controller E2E on OpenShift ---------------------------------------- Checking prerequisites... Prerequisites met — logged in as: system:admin DEPLOY_MODE: kustomize MAAS_API_IMAGE: quay.io/opendatahub/maas-api:latest MAAS_CONTROLLER_IMAGE: quay.io/opendatahub/maas-controller:latest AI_GATEWAY_CONTROLLER_IMAGE: quay.io/opendatahub/odh-ai-gateway-controller@sha256:fd19843e47bfe605ec44d877133464e5b56e3298fae56226e9eaaae0f07c1dc6 PRAXIS_EXTPROC_IMAGE: quay.io/opendatahub/odh-praxis-extproc:odh-stable 2026-09-16T14:12:47Z deploy_platform start Deploying MaaS platform via ODH operator... Gateway ingress mode for deploy.sh: ocproute (loadbalancer | ocproute) Using custom MaaS API image: quay.io/opendatahub/maas-api:latest Using custom MaaS controller image: quay.io/opendatahub/maas-controller:latest Deployment mode: kustomize Installing cert-manager and LeaderWorkerSet operators... === Installing cert-manager and LeaderWorkerSet operators === 1. Installing cert-manager operator... namespace/cert-manager-operator created operatorgroup.operators.coreos.com/cert-manager-operator created subscription.operators.coreos.com/openshift-cert-manager-operator created Waiting for Subscription cert-manager-operator/openshift-cert-manager-operator... subscription.operators.coreos.com/openshift-cert-manager-operator condition met Waiting for CSV cert-manager-operator.v1.20.0 to succeed... clusterserviceversion.operators.coreos.com/cert-manager-operator.v1.20.0 condition met cert-manager ready. 2. Installing LeaderWorkerSet operator... namespace/openshift-lws-operator created operatorgroup.operators.coreos.com/leader-worker-set created subscription.operators.coreos.com/leader-worker-set created Waiting for Subscription openshift-lws-operator/leader-worker-set... subscription.operators.coreos.com/leader-worker-set condition met Waiting for CSV leader-worker-set.v1.0.0 to succeed... clusterserviceversion.operators.coreos.com/leader-worker-set.v1.0.0 condition met LeaderWorkerSet operator ready. 3. Activating LeaderWorkerSet API... leaderworkersetoperator.operator.openshift.io/cluster created LeaderWorkerSetOperator CR applied. === Done === Verify: oc get pods -n cert-manager-operator oc get pods -n openshift-lws-operator oc get crd leaderworkersets.leaderworkerset.x-k8s.io Installing OpenDataHub operator... === Installing OpenDataHub operator === 1. Setting up ODH catalog... Using community-operators 2. Installing ODH operator... [INFO] Installing operator: opendatahub-operator in namespace: opendatahub [INFO] Creating namespace: opendatahub namespace/opendatahub created namespace/opendatahub condition met [INFO] Creating OperatorGroup in opendatahub for AllNamespaces mode operatorgroup.operators.coreos.com/opendatahub-operatorgroup created [INFO] Creating Subscription for opendatahub-operator from community-operators (channel: fast-3, installPlanApproval: Manual, startingCSV: opendatahub-operator.v3.5.0-ea.2) subscription.operators.coreos.com/opendatahub-operator created [INFO] Manual Subscription: approving initial InstallPlan so first install can proceed... [INFO] Approving initial InstallPlan install-9t5cw (Manual subscription) installplan.operators.coreos.com/install-9t5cw patched [INFO] Waiting for subscription to install... * Waiting for Subscription opendatahub/opendatahub-operator to start setup... subscription.operators.coreos.com/opendatahub-operator condition met * Waiting for Subscription setup to finish setup. CSV = opendatahub-operator.v3.5.0-ea.2 ... clusterserviceversion.operators.coreos.com/opendatahub-operator.v3.5.0-ea.2 condition met [INFO] Operator opendatahub-operator installed successfully 3. Skipping operator image patch (OPERATOR_IMAGE not set) 4. Waiting for operator CRDs... ⏳ Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 180s)… ✅ CRD datascienceclusters.datasciencecluster.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met 5. Waiting for operator webhook... * Waiting for deployment/opendatahub-operator-controller-manager in opendatahub (timeout: 120s)... * Found deployment/opendatahub-operator-controller-manager deployment.apps/opendatahub-operator-controller-manager condition met 6. Applying DSCInitialization... dscinitialization.dscinitialization.opendatahub.io/default-dsci created Waiting for DSCInitialization to be Ready... Waiting for DSCInitialization Ready (attempt 1/30, phase=Progressing, Ready=unknown)... DSCInitialization is Ready 7. Applying DataScienceCluster... datasciencecluster.datasciencecluster.opendatahub.io/default-dsc serverside-applied 8. Waiting for DataScienceCluster (KServe)... * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... - KServe state: , KserveReady: , ModelsAsServiceReady: (informational only), AIGatewayReady: - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: False (informational only), AIGatewayReady: - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: False (informational only), AIGatewayReady: * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' 9. Waiting for odh-model-controller webhook... ⏳ Waiting for validating webhooks in namespace opendatahub (timeout: 180s)... ✅ Webhook service opendatahub/kserve-webhook-server-service has ready endpoints ✅ Webhook service opendatahub/llmisvc-webhook-server-service has ready endpoints ✅ Webhook service opendatahub/odh-model-controller-webhook-service has ready endpoints ✅ Webhook service opendatahub/opendatahub-operator-controller-manager-service has ready endpoints 🎉 All validating webhook services in opendatahub are ready === ODH installation complete === Verify: kubectl get datasciencecluster -A kubectl get pods -n opendatahub kubectl get pods -n kserve Using policy engine: rhcl (Authorino namespace: kuadrant-system) [INFO] =================================================== [INFO] Models-as-a-Service Deployment [INFO] =================================================== [INFO] Validating configuration... [INFO] Configuration validated successfully [INFO] Deployment configuration: [INFO] Mode: kustomize [INFO] Policy Engine: rhcl [INFO] Namespace: opendatahub [INFO] TLS Backend: true [INFO] External OIDC: false [INFO] MaaS API image: quay.io/opendatahub/maas-api:latest [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:latest [INFO] Starting kustomize-based deployment... [INFO] Installing policy engine: rhcl [INFO] Installing RHCL (Red Hat Connectivity Link - downstream) [INFO] Using RHCL channel head from redhat-operators (stable) [INFO] Installing RHCL into namespace: kuadrant-system [INFO] Installing operator: rhcl-operator in namespace: kuadrant-system [INFO] Creating namespace: kuadrant-system namespace/kuadrant-system created namespace/kuadrant-system condition met [INFO] Creating OperatorGroup in kuadrant-system for AllNamespaces mode operatorgroup.operators.coreos.com/kuadrant-system-operatorgroup created [INFO] Creating Subscription for rhcl-operator from redhat-operators (channel: stable) subscription.operators.coreos.com/rhcl-operator created [INFO] Waiting for subscription to install... * Waiting for Subscription kuadrant-system/rhcl-operator to start setup... subscription.operators.coreos.com/rhcl-operator condition met * Waiting for Subscription setup to finish setup. CSV = rhcl-operator.v1.4.3 ... clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 condition met [INFO] Operator rhcl-operator installed successfully [INFO] Patching rhcl-operator CSV (Gateway API, rate limit failure modes, auth service timeout)... clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched [INFO] CSV patched (Gateway controller and/or rate limit failure modes and/or auth timeout) [INFO] Forcing operator restart to apply CSV env configuration... pod "kuadrant-operator-controller-manager-67c5dcb48c-zccsh" force deleted pod "kuadrant-operator-controller-manager-76f8f686bf-29hwx" force deleted pod "limitador-operator-controller-manager-74495c69f8-qg7pp" force deleted [INFO] Waiting for operator pod to restart... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... deployment "kuadrant-operator-controller-manager" successfully rolled out [WARN] Operator pod may not have correct env yet (ISTIO / RATELIMIT_* failure modes / AUTH_SERVICE_TIMEOUT) [INFO] Waiting 15s for operator to fully initialize with Gateway controller configuration... [INFO] Initializing Gateway API and ModelsAsAService gateway... [INFO] =================================================== [INFO] MaaS Gateway Setup [INFO] =================================================== [INFO] Validating gateway configuration... [INFO] Configuration validated [INFO] Ingress mode: ocproute [INFO] Disconnected: false [INFO] AllowedRoutes: namespaces=opendatahub,odh-ai-gateway-infra,llm [INFO] Detecting cluster domain... [INFO] Detected cluster domain: apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com [INFO] Setting up GatewayClass... [INFO] Creating GatewayClass openshift-default... gatewayclass.gateway.networking.k8s.io/openshift-default created [INFO] Setting up Gateway in ocproute mode (ClusterIP with OpenShift Route)... [INFO] Creating ConfigMap gw-options... configmap/gw-options created [INFO] Creating/updating Gateway maas-default-gateway (ocproute mode)... gateway.gateway.networking.k8s.io/maas-default-gateway serverside-applied [INFO] Waiting for Gateway to be Programmed (timeout: 120s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met [INFO] Gateway is Programmed [INFO] Gateway Service ready: maas-default-gateway-openshift-default [INFO] Creating Route maas-gateway-route... [INFO] Host: maas.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com [INFO] Target Service: maas-default-gateway-openshift-default route.route.openshift.io/maas-gateway-route created [INFO] Waiting for Route to be Admitted... [INFO] Route is Admitted [INFO] ClusterIP mode setup complete [INFO] [INFO] =================================================== [INFO] Gateway setup completed successfully [INFO] =================================================== [INFO] Mode: ocproute [INFO] Gateway: maas-default-gateway [INFO] Namespace: openshift-ingress [INFO] Route: maas-gateway-route [INFO] Hostname: maas.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com [INFO] [INFO] Verify with: [INFO] kubectl get gateway maas-default-gateway -n openshift-ingress [INFO] kubectl get route maas-gateway-route -n openshift-ingress [INFO] Applying Kuadrant custom resource in kuadrant-system... kuadrant.kuadrant.io/kuadrant created [INFO] Waiting for Kuadrant to become ready (initial check)... [INFO] Waiting for: Kuadrant ready in kuadrant-system (timeout: 60s) [INFO] Kuadrant ready in kuadrant-system - Ready [INFO] Kuadrant setup complete [INFO] Ensuring namespace exists: opendatahub [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [WARN] DEPLOYING POC POSTGRESQL — NOT INTENDED FOR PRODUCTION USE [WARN] Data is stored in ephemeral storage and will be lost on pod restart. [WARN] For production, use --postgres-connection with an external database [WARN] (AWS RDS, Crunchy Operator, Azure Database, etc.) [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ⚠️ WARNING FOR PRODUCTION USE. ⚠️ ┃ ┃ This deploys PostgreSQL with ephemeral storage (emptyDir). ┃ ┃ Data WILL be lost on pod restart. ┃ ┃ For production, use an external database: ┃ ┃ deploy.sh --postgres-connection postgresql://... ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 🔧 Fresh install: Deploying PostgreSQL in infrastructure namespace 'odh-ai-gateway-infra'... 📦 Creating infrastructure namespace 'odh-ai-gateway-infra'... namespace/odh-ai-gateway-infra created Generated random PostgreSQL password (stored in secret postgres-creds) Creating PostgreSQL deployment... ⚠️ Using POC configuration (ephemeral storage) Using default PostgreSQL image (operator CSV not available) Image: registry.redhat.io/rhel9/postgresql-16:latest secret/postgres-creds created deployment.apps/postgres created service/postgres created secret/maas-db-config created Waiting for PostgreSQL to be ready... deployment.apps/postgres condition met ✅ PostgreSQL deployed successfully Namespace: odh-ai-gateway-infra Database: maas User: maas Secret: maas-db-config (contains DB_CONNECTION_URL with FQDN) ⚠️ For production, use AWS RDS, Crunchy Operator, or Azure Database Note: Schema migrations run automatically when maas-api starts [INFO] Configuring TLS backend for Authorino and MaaS API... * Waiting for deployment/authorino in kuadrant-system (timeout: 300s)... * Found deployment/authorino [INFO] Running TLS configuration script... [INFO] TLS configuration script completed successfully [INFO] Restarting deployments to pick up TLS configuration... deployment.apps/authorino restarted [INFO] Waiting for Authorino deployment to be ready... Waiting for deployment spec update to be observed... Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out [INFO] TLS backend configuration complete [INFO] Kustomize prerequisite deployment completed [INFO] [INFO] MaaS Controller... [INFO] Phase 1: Applying MaaS CRDs and waiting until Established (controller creates Config after CRD is ready)... ⏳ Applying MaaS Controller CRDs from /workspace/source/test/maas-e2e/scripts/../deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io created ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] Phase 2: Applying full controller kustomize (same as operator: deployment/base/maas-controller/default)... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io configured serviceaccount/maas-controller created role.rbac.authorization.k8s.io/maas-controller-leader-election-role created clusterrole.rbac.authorization.k8s.io/maas-controller-cluster-config-role created clusterrole.rbac.authorization.k8s.io/maas-controller-ocp-role created clusterrole.rbac.authorization.k8s.io/maas-controller-role created clusterrole.rbac.authorization.k8s.io/maas-owner-role created clusterrole.rbac.authorization.k8s.io/maas-viewer-role created rolebinding.rbac.authorization.k8s.io/maas-controller-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-cluster-config-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-ocp-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-rolebinding created configmap/maas-parameters created service/maas-controller-metrics created service/maas-controller-webhook-service created deployment.apps/maas-controller created servicemonitor.monitoring.coreos.com/maas-controller-metrics created networkpolicy.networking.k8s.io/maas-controller-allow-monitoring created validatingwebhookconfiguration.admissionregistration.k8s.io/maas-validating-webhook-configuration created [INFO] Restarting maas-controller to pick up manifest and ConfigMap changes deployment.apps/maas-controller restarted [INFO] Waiting for maas-controller to be ready... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... deployment "maas-controller" successfully rolled out [INFO] Controller ready. [INFO] [INFO] Waiting for Tenant reconciler to deploy maas-api... [INFO] Applying secret migration RBAC to namespace odh-ai-gateway-infra... role.rbac.authorization.k8s.io/maas-controller-secret-migrate created rolebinding.rbac.authorization.k8s.io/maas-controller-secret-migrate created [INFO] maas-api deployment found in odh-ai-gateway-infra, waiting for rollout... deployment "maas-api" successfully rolled out [INFO] maas-api is ready [INFO] [INFO] MaaS API and MaaS Controller deployment completed successfully! [INFO] maas-api image: quay.io/opendatahub/maas-api:latest (namespace: odh-ai-gateway-infra) [INFO] maas-controller image: quay.io/opendatahub/maas-controller:latest (namespace: opendatahub) [INFO] =================================================== [INFO] Models-as-a-Service Deployment completed successfully! [INFO] =================================================== * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' ⚠️ WARNING: Skipping Authorino readiness check (SKIP_AUTH_CHECK=true) ✅ MaaS platform deployment completed 2026-09-16T14:18:26Z deploy_platform end ---------------------------------------- Deploying Models ---------------------------------------- 2026-09-16T14:18:26Z deploy_models start Deploying MaaS system (free + premium: LLMIS + MaaSModelRef + MaaSAuthPolicy + MaaSSubscription) Waiting for Gateway openshift-ingress/maas-default-gateway to be Programmed=True (timeout: 600s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met ✅ Gateway openshift-ingress/maas-default-gateway is Programmed Creating 'llm' namespace... namespace/llm created 'models-as-a-service' namespace already exists maasauthpolicy.maas.opendatahub.io/premium-simulator-access created maasauthpolicy.maas.opendatahub.io/simulator-access created maasmodelref.maas.opendatahub.io/e2e-distinct-2-simulated created maasmodelref.maas.opendatahub.io/e2e-distinct-simulated created maasmodelref.maas.opendatahub.io/e2e-embedding-simulated created maasmodelref.maas.opendatahub.io/e2e-trlp-test-simulated created maasmodelref.maas.opendatahub.io/e2e-unconfigured-facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/premium-simulated-simulated-premium created maassubscription.maas.opendatahub.io/premium-simulator-subscription created maassubscription.maas.opendatahub.io/simulator-subscription created llminferenceservice.serving.kserve.io/e2e-distinct-2-simulated created llminferenceservice.serving.kserve.io/e2e-distinct-simulated created llminferenceservice.serving.kserve.io/e2e-embedding-simulated created llminferenceservice.serving.kserve.io/e2e-trlp-test-simulated created llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium created ✅ MaaS system deployed (free + premium + e2e test fixtures) Waiting for models to be ready (timeout: 300s)... llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated condition met llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium condition met llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated condition met ✅ Simulator models ready Waiting for governed MaaSModelRefs to be Ready (timeout: 300s)... ✅ Governed MaaSModelRefs ready Waiting for MaaS Kuadrant AuthPolicies to be enforced (selector: app.kubernetes.io/managed-by=maas-controller, timeout: 180s)... ✅ All MaaS AuthPolicies enforced (1 policies) 2026-09-16T14:20:58Z deploy_models end Patching Authorino to log_level DEBUG... authorino.operator.authorino.kuadrant.io/authorino patched ✅ Authorino patched to log_level DEBUG Waiting for deployment spec update to be observed... Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out ---------------------------------------- Deploying ai-gateway-controller ---------------------------------------- 2026-09-16T14:21:01Z deploy_ai_gateway_controller start Deploying ai-gateway-controller image: quay.io/opendatahub/odh-ai-gateway-controller@sha256:fd19843e47bfe605ec44d877133464e5b56e3298fae56226e9eaaae0f07c1dc6 praxis-extproc image: quay.io/opendatahub/odh-praxis-extproc:odh-stable namespace: opendatahub gateway: openshift-ingress/maas-default-gateway remove maas IPP: true Opting default AITenant into praxis dataplane (maas.opendatahub.io/payload-processing-type=praxis) ... aitenant.maas.opendatahub.io/models-as-a-service annotated Pausing maas-controller (scale 1 -> 0) to avoid IPP reconcile during handoff ... deployment.apps/maas-controller scaled deployment "maas-controller" successfully rolled out Removing maas-controller legacy IPP in openshift-ingress ... deployment.apps "payload-processing" deleted service "payload-processing" deleted destinationrule.networking.istio.io "payload-processing" deleted deployment.apps "payload-pre-processing" deleted service "payload-pre-processing" deleted destinationrule.networking.istio.io "payload-pre-processing" deleted envoyfilter.networking.istio.io "payload-processing" deleted networkpolicy.networking.k8s.io "payload-processing" deleted No resources found Legacy IPP Deployments removed from openshift-ingress Resuming maas-controller so ai-gateway-controller can reconcile AITenant (webhook) ... Resuming maas-controller (scale -> 1) ... deployment.apps/maas-controller scaled Waiting for deployment "maas-controller" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "maas-controller" rollout to finish: 0 of 1 updated replicas are available... deployment "maas-controller" successfully rolled out serviceaccount/ai-gateway-controller created clusterrole.rbac.authorization.k8s.io/ai-gateway-controller-role created clusterrolebinding.rbac.authorization.k8s.io/ai-gateway-controller-rolebinding created configmap/ai-gateway-controller-parameters created deployment.apps/ai-gateway-controller created Waiting for deployment "ai-gateway-controller" rollout to finish: 0 of 1 updated replicas are available... deployment "ai-gateway-controller" successfully rolled out Triggering immediate praxis-extproc install ... deployment.apps/ai-gateway-controller restarted Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... deployment "ai-gateway-controller" successfully rolled out Waiting for ai-gateway-controller to attach praxis finalizer on ai-tenants/models-as-a-service ... ai-gateway-controller praxis reconcile started (finalizer present) Waiting for praxis-extproc (quay.io/opendatahub/odh-praxis-extproc:odh-stable) in openshift-ingress (timeout: 300s) ... ✅ praxis-extproc ready: openshift-ingress/payload-processing image=quay.io/opendatahub/odh-praxis-extproc:odh-stable Annotating praxis IPP resources opendatahub.io/managed=false so maas-controller skips them ... deployment.apps/payload-processing annotated service/payload-processing annotated destinationrule.networking.istio.io/payload-processing annotated deployment.apps/payload-pre-processing annotated service/payload-pre-processing annotated destinationrule.networking.istio.io/payload-pre-processing annotated envoyfilter.networking.istio.io/payload-processing annotated networkpolicy.networking.k8s.io/payload-processing annotated Verified ext_proc dataplane image: quay.io/opendatahub/odh-praxis-extproc:odh-stable ai-gateway-controller rollout complete (praxis-extproc handoff done) 2026-09-16T14:21:47Z deploy_ai_gateway_controller end ---------------------------------------- Enabling tenant namespace discovery ---------------------------------------- 2026-09-16T14:21:47Z tenant_namespace_discovery start Enabling --enable-tenant-namespace-discovery on maas-controller... maas-controller already has tenant namespace discovery enabled 2026-09-16T14:21:48Z tenant_namespace_discovery end ---------------------------------------- Setting up variables for tests ---------------------------------------- -- Setting up variables for tests -- HOST: maas.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com MAAS_API_BASE_URL: https://maas.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com/maas-api ---------------------------------------- Setting up test tokens ---------------------------------------- Setting up premium test token (SA-based, works when oc whoami -t is unavailable)... Creating namespace: premium-users-namespace namespace/premium-users-namespace created Creating service account: premium-service-account serviceaccount/premium-service-account created Patching MaaSAuthPolicy premium-simulator-access to include system:serviceaccount:premium-users-namespace:premium-service-account... maasauthpolicy.maas.opendatahub.io/premium-simulator-access patched Patching MaaSSubscription premium-simulator-subscription to include system:serviceaccount:premium-users-namespace:premium-service-account... maassubscription.maas.opendatahub.io/premium-simulator-subscription patched Waiting for MaaSSubscriptions to reconcile after patch (timeout: 60s)... ✅ Both subscriptions ready: simulator-subscription=Active, premium-simulator-subscription=Active ✅ Premium test token setup complete (E2E_TEST_TOKEN_SA_* exported) Setting up test tokens (admin + regular user)... Current admin session: system:admin (will be preserved) ⚠️ No htpasswd token available - using SA token (admin tests may fail) Creating namespace: maas-admin namespace/maas-admin created Creating service account: tester-admin-user in maas-admin serviceaccount/tester-admin-user created Creating cluster role binding for tester-admin-user clusterrolebinding.rbac.authorization.k8s.io/tester-admin-user-binding created ✅ User setup completed: tester-admin-user (namespace: maas-admin) role.rbac.authorization.k8s.io/maas-admin-e2e created rolebinding.rbac.authorization.k8s.io/maas-admin-e2e-system-serviceaccount-maas-admin-tester-admin-user created auth.services.platform.opendatahub.io/auth patched ✅ Added system:serviceaccounts:maas-admin to Auth CR adminGroups (SA admin fallback) clusterrole.rbac.authorization.k8s.io/maas-admin created rolebinding.rbac.authorization.k8s.io/odh-admins-maas-admin created Creating separate SA token for regular user (required for IDOR tests)... Creating service account: tester-regular-user in default serviceaccount/tester-regular-user created Creating cluster role binding for tester-regular-user clusterrolebinding.rbac.authorization.k8s.io/tester-regular-user-binding created ✅ User setup completed: tester-regular-user (namespace: default) ✅ Regular user token for tester-regular-user (SA-based, namespace: default) Token setup complete (main session unchanged: system:admin) ---------------------------------------- Validating Deployment ---------------------------------------- 2026-09-16T14:21:52Z validate start Deployment Validation ========================================= 🚀 MaaS Platform Deployment Validation ========================================= ========================================= 1️⃣ Component Status Checks ========================================= 🔍 Checking: MaaS API pods ✅ PASS: MaaS API has 1 running pod(s) 🔍 Checking: Policy engine pods (RHCL/Kuadrant) ✅ PASS: Policy engine has 7 running pod(s) in kuadrant-system 🔍 Checking: OpenDataHub/KServe pods ℹ️ opendatahub namespace: 9 running pod(s) ✅ PASS: OpenDataHub/RHOAI has 9 total running pod(s) 🔍 Checking: LLM namespace and models ✅ PASS: Found 7 LLMInferenceService(s) with 7 running pod(s) ========================================= 2️⃣ Gateway Status ========================================= 🔍 Checking: Gateway resource ✅ PASS: Gateway is Accepted and Programmed 🔍 Checking: HTTPRoute for maas-api ✅ PASS: HTTPRoute maas-api-route is configured and accepted 🔍 Checking: Gateway hostname ✅ PASS: Gateway hostname: https://maas.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com ========================================= 3️⃣ Policy Status ========================================= 🔍 Checking: AuthPolicy ✅ PASS: AuthPolicy is configured and accepted 🔍 Checking: TokenRateLimitPolicy ✅ PASS: TokenRateLimitPolicy is configured and accepted ========================================= 4️⃣ API Endpoint Tests ========================================= ℹ️ Using gateway endpoint: https://maas.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com 🔍 Checking: Authentication token ✅ PASS: OpenShift identity token available 🔍 Checking: MaaS API key creation ✅ PASS: MaaS API key created (name: validate-test-1789568514) 🔍 Checking: Models endpoint ℹ️ Testing: curl -sSk https://maas.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com/maas-api/v1/models -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" ✅ PASS: Models endpoint accessible, found 1 model(s) ℹ️ Available models: • publishers/llm/models/facebook/opt-125m - https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ ℹ️ Using first available model: publishers/llm/models/facebook/opt-125m for validation ℹ️ Rewrote internal model URL to external gateway: https://maas.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com/ 🔍 Checking: Model inference endpoint ℹ️ Testing: curl -sSk -X POST https://maas.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com/v1/chat/completions -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"model": "publishers/llm/models/facebook/opt-125m", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50}' ✅ PASS: Model inference endpoint working ℹ️ Response: {"id":"chatcmpl-b8479eb1-ec12-5f57-ac58-aa5ea7d18646","created":1789568515,"model":"facebook/opt-125m","usage":{"prompt_tokens":6,"completion_tokens":50,"total_tokens":56},"object":"chat.completion"," 🔍 Checking: Rate limiting ℹ️ User tier: unknown (could not extract from token) ℹ️ Sending 10 rapid requests to test rate limiting... ✅ PASS: Rate limiting is working (3 successful, 7 rate limited) 🔍 Checking: Authorization enforcement (401 without token) ✅ PASS: Authorization is enforced (got 401 without token) ========================================= 📊 Validation Summary ========================================= Results: ✅ Passed: 15 ❌ Failed: 0 ⚠️ Warnings: 0 ✅ PASS: All critical checks passed! 🎉 Next steps: 1. Deploy a model: kustomize build docs/samples/models/simulator | kubectl apply -f - 2. Access the API at: https://maas.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com} 3. Check documentation: docs/README.md 4. Re-run validation with specific model: ./scripts/validate-deployment.sh MODEL_NAME Deployment validation completed 2026-09-16T14:21:56Z validate end ---------------------------------------- Running E2E Tests ---------------------------------------- -- E2E Tests (ai-gateway-controller, no external-model tests) -- Waiting for gateway: https://maas.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com/maas-api/health ... Gateway reachable (HTTP 200) Waiting for authenticated gateway access ... Authenticated gateway access working (HTTP 200) Creating Python venv for e2e tests... WARNING: The directory '/opt/app-root/src/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. Requirement already satisfied: pip in ./test/e2e/.venv/lib/python3.9/site-packages (21.3.1) Collecting pip Downloading pip-26.0.1-py3-none-any.whl (1.8 MB) Requirement already satisfied: setuptools in ./test/e2e/.venv/lib/python3.9/site-packages (53.0.0) Collecting setuptools Downloading setuptools-82.0.1-py3-none-any.whl (1.0 MB) Installing collected packages: setuptools, pip Attempting uninstall: setuptools Found existing installation: setuptools 53.0.0 Uninstalling setuptools-53.0.0: Successfully uninstalled setuptools-53.0.0 Attempting uninstall: pip Found existing installation: pip 21.3.1 Uninstalling pip-21.3.1: Successfully uninstalled pip-21.3.1 Successfully installed pip-26.0.1 setuptools-82.0.1 WARNING: The directory '/opt/app-root/src/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. WARNING: The directory '/opt/app-root/src/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. Running E2E pass 1/2: parallel (E2E_PARALLEL_WORKERS=7, --dist=loadgroup, -m 'not serial') ============================= test session starts ============================== platform linux -- Python 3.9.25, pytest-8.4.2, pluggy-1.6.0 -- /workspace/source/test/e2e/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.9.25', 'Platform': 'Linux-5.14.0-570.134.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '8.4.2', 'pluggy': '1.6.0'}, 'Plugins': {'xdist': '3.8.0', 'metadata': '3.1.1', 'html': '4.2.0'}, 'PLATFORM': 'el9'} rootdir: /workspace/source/test/maas-e2e/test/e2e configfile: pyproject.toml plugins: xdist-3.8.0, metadata-3.1.1, html-4.2.0 created: 7/7 workers 7 workers [210 items] scheduling tests via LoadGroupScheduling test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_default_aitenant_bootstraps_maas_tenant_config_without_gateway_mutation@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_model_routes_through_tenant_gateway@tenant_isolation test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_explicit_subscription_header@models test/maas-e2e/test/e2e/tests/test_smoke.py::test_healthz_or_404@readonly test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_subscription_namespace_visible_to_api@security test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_create_api_key@api_keys [gw4] [ 0%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_subscription_namespace_visible_to_api@security test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_another_namespace_not_visible_to_api@security [gw4] [ 0%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_another_namespace_not_visible_to_api@security test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_maas_subscription_namespace@security [gw4] [ 1%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_maas_subscription_namespace@security test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_another_namespace@security [gw4] [ 1%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_another_namespace@security test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_auth_policy_model_ref@security [gw4] [ 2%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_auth_policy_model_ref@security test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_subscription_model_ref@security [gw4] [ 2%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_subscription_model_ref@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAPIKeyManagementIsolation::test_api_key_cannot_mint_another_api_key@security [gw5] [ 3%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_healthz_or_404@readonly test/maas-e2e/test/e2e/tests/test_smoke.py::test_tokens_endpoint_replaced_by_api_keys@readonly [gw5] [ 3%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_tokens_endpoint_replaced_by_api_keys@readonly test/maas-e2e/test/e2e/tests/test_smoke.py::test_models_catalog@readonly [gw5] [ 4%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_models_catalog@readonly test/maas-e2e/test/e2e/tests/test_smoke.py::test_chat_completions_gateway_alive@readonly [gw4] [ 4%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAPIKeyManagementIsolation::test_api_key_cannot_mint_another_api_key@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[username-only]@security [gw5] [ 5%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_chat_completions_gateway_alive@readonly test/maas-e2e/test/e2e/tests/test_smoke.py::test_legacy_completions_optionally@readonly [gw5] [ 5%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_legacy_completions_optionally@readonly test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_tenant_ready_and_phase_healthy@readonly [gw4] [ 6%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[username-only]@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[group-only]@security [gw4] [ 6%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[group-only]@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_injected_identity_headers_rejected_on_inference@security [gw4] [ 7%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_injected_identity_headers_rejected_on_inference@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_duplicate_subscription_headers_ignored@security [gw4] [ 7%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_duplicate_subscription_headers_ignored@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestExpiredKeyRejection::test_expired_key_rejected_at_gateway@security [gw4] [ 8%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestExpiredKeyRejection::test_expired_key_rejected_at_gateway@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestCrossModelAccess::test_key_cannot_access_model_outside_subscription@security [gw4] [ 8%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestCrossModelAccess::test_key_cannot_access_model_outside_subscription@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_subscription_with_nonexistent_model_ref@security [gw4] [ 9%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_subscription_with_nonexistent_model_ref@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_authpolicy_with_nonexistent_model_ref@security [gw4] [ 9%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_authpolicy_with_nonexistent_model_ref@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderAbuse::test_special_characters_in_subscription_header@security [gw4] [ 10%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderAbuse::test_special_characters_in_subscription_header@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestWebhookValidation::test_subscription_rejected_in_unlabeled_namespace@security [gw4] [ 10%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestWebhookValidation::test_subscription_rejected_in_unlabeled_namespace@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestWebhookValidation::test_authpolicy_rejected_in_unlabeled_namespace@security [gw4] [ 10%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestWebhookValidation::test_authpolicy_rejected_in_unlabeled_namespace@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[subscriptions-select]@security [gw4] [ 11%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[subscriptions-select]@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-cleanup]@security [gw4] [ 11%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-cleanup]@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-validate]@security [gw4] [ 12%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-validate]@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_health_endpoint_accessible@security [gw4] [ 12%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_health_endpoint_accessible@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_v1_models_via_maas_api_prefix@security [gw4] [ 13%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_v1_models_via_maas_api_prefix@security [gw0] [ 13%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_create_api_key@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_list_api_keys@api_keys [gw0] [ 14%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_list_api_keys@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_revoke_api_key@api_keys [gw0] [ 14%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_revoke_api_key@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_admin_manage_other_users_keys@api_keys [gw2] [ 15%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_model_routes_through_tenant_gateway@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_inference_succeeds_through_tenant_gateway@tenant_isolation [gw0] [ 15%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_admin_manage_other_users_keys@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_non_admin_cannot_access_other_users_keys@api_keys [gw0] [ 16%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_non_admin_cannot_access_other_users_keys@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_own_keys@api_keys [gw0] [ 16%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_own_keys@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_other_user_forbidden@api_keys [gw0] [ 17%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_other_user_forbidden@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_admin_can_revoke_any_user@api_keys [gw0] [ 17%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_admin_can_revoke_any_user@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_by_subscription@api_keys [gw0] [ 18%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_by_subscription@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_by_subscription_forbidden_for_non_admin@api_keys [gw0] [ 18%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_by_subscription_forbidden_for_non_admin@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_dry_run@api_keys [gw0] [ 19%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_dry_run@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_dry_run_by_subscription@api_keys [gw0] [ 19%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_dry_run_by_subscription@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_combined_user_and_subscription@api_keys [gw0] [ 20%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_combined_user_and_subscription@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_missing_scope_returns_400@api_keys [gw0] [ 20%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_missing_scope_returns_400@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_create_key_within_expiration_limit@api_keys [gw0] [ 20%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_create_key_within_expiration_limit@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_create_key_at_expiration_limit@api_keys [gw0] [ 21%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_create_key_at_expiration_limit@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_create_key_exceeds_expiration_limit@api_keys [gw0] [ 21%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_create_key_exceeds_expiration_limit@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_create_key_without_expiration@api_keys [gw0] [ 22%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_create_key_without_expiration@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_create_key_with_short_expiration@api_keys [gw0] [ 22%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_create_key_with_short_expiration@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_api_key_model_access_success@api_keys [gw0] [ 23%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_api_key_model_access_success@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_invalid_api_key_rejected@api_keys [gw0] [ 23%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_invalid_api_key_rejected@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_no_auth_header_rejected@api_keys [gw0] [ 24%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_no_auth_header_rejected@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_revoked_api_key_rejected@api_keys [gw2] [ 24%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_inference_succeeds_through_tenant_gateway@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_tenant_isolation_cross_gateway_blocked@tenant_isolation [gw2] [ 25%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_tenant_isolation_cross_gateway_blocked@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_correct_model_in_body_succeeds@tenant_isolation [gw0] [ 25%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_revoked_api_key_rejected@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_api_key_chat_completions@api_keys [gw0] [ 26%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_api_key_chat_completions@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_double_revoke_returns_404@api_keys [gw0] [ 26%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_double_revoke_returns_404@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_nonexistent_key_returns_404@api_keys [gw0] [ 27%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_nonexistent_key_returns_404@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_then_create_new_key_works@api_keys [gw0] [ 27%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_then_create_new_key_works@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_individual_revoke_multiple_keys@api_keys [gw0] [ 28%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_individual_revoke_multiple_keys@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_keys_rejected_at_gateway@api_keys [gw0] [ 28%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_keys_rejected_at_gateway@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_cronjob_exists_and_configured@api_keys [gw0] [ 29%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_cronjob_exists_and_configured@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_cleanup_networkpolicy_exists@api_keys [gw0] [ 29%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_cleanup_networkpolicy_exists@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_create_ephemeral_key@api_keys [gw0] [ 30%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_create_ephemeral_key@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_trigger_cleanup_preserves_active_keys@api_keys [gw0] [ 30%] SKIPPED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_trigger_cleanup_preserves_active_keys@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_active_subscription@api_keys [gw0] [ 30%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_active_subscription@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_degraded_subscription@api_keys [gw5] [ 31%] FAILED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_tenant_ready_and_phase_healthy@readonly test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_status_has_phase_and_conditions@readonly [gw5] [ 31%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_status_has_phase_and_conditions@readonly test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_spec_is_well_formed@readonly [gw0] [ 32%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_degraded_subscription@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_failed_subscription@api_keys [gw5] [ 32%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_spec_is_well_formed@readonly test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_conditions_use_kubernetes_metav1_shape@readonly [gw5] [ 33%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_conditions_use_kubernetes_metav1_shape@readonly test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantNoFalseOwnership::test_maas_user_crs_not_owned_by_tenant@readonly [gw5] [ 33%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantNoFalseOwnership::test_maas_user_crs_not_owned_by_tenant@readonly test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_default_exists@readonly [gw5] [ 34%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_default_exists@readonly test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_not_terminating@readonly [gw5] [ 34%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_not_terminating@readonly test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_default_aitenant_lists_config_owner_reference@readonly [gw5] [ 35%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_default_aitenant_lists_config_owner_reference@readonly test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_tenant_config_lists_config_owner_reference@readonly [gw5] [ 35%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_tenant_config_lists_config_owner_reference@readonly test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_maas_controller_deployment_does_not_list_config_owner_reference@readonly [gw5] [ 36%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_maas_controller_deployment_does_not_list_config_owner_reference@readonly test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_requires_auth@readonly [gw0] [ 36%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_failed_subscription@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionFilter::test_search_filters_by_subscription@api_keys [gw2] [ 37%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_correct_model_in_body_succeeds@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_wrong_model_in_body_rejected@tenant_isolation [gw0] [ 37%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionFilter::test_search_filters_by_subscription@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionFilter::test_search_without_subscription_returns_all@api_keys [gw0] [ 38%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionFilter::test_search_without_subscription_returns_all@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_create_api_key_with_labels@api_keys [gw0] [ 38%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_create_api_key_with_labels@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_search_api_keys_by_labels@api_keys [gw0] [ 39%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_search_api_keys_by_labels@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_labels_validation_errors@api_keys [gw0] [ 39%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_labels_validation_errors@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_backward_compatibility_no_labels@api_keys [gw0] [ 40%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_backward_compatibility_no_labels@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_authorized_user_gets_200@api_keys [gw0] [ 40%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_authorized_user_gets_200@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_no_auth_gets_401@api_keys [gw0] [ 40%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_no_auth_gets_401@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_invalid_token_gets_403@api_keys [gw0] [ 41%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_invalid_token_gets_403@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_wrong_group_gets_403@api_keys [gw0] [ 41%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_wrong_group_gets_403@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_uses_highest_priority_subscription@api_keys [gw0] [ 42%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_uses_highest_priority_subscription@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_with_explicit_simulator_subscription@api_keys [gw0] [ 42%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_with_explicit_simulator_subscription@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_nonexistent_subscription_errors@api_keys [gw0] [ 43%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_nonexistent_subscription_errors@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_subscribed_user_gets_200@api_keys [gw5] [ 43%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_requires_auth@readonly test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_with_invalid_token@readonly [gw0] [ 44%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_subscribed_user_gets_200@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_auth_pass_no_subscription_gets_403@api_keys [gw0] [ 44%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_auth_pass_no_subscription_gets_403@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestMultipleAuthPoliciesPerModel::test_two_auth_policies_or_logic@api_keys [gw0] [ 45%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestMultipleAuthPoliciesPerModel::test_two_auth_policies_or_logic@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestOrderingEdgeCases::test_subscription_before_auth_policy@api_keys [gw5] [ 45%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_with_invalid_token@readonly test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_authenticated@readonly [gw5] [ 46%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_authenticated@readonly test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_gateway_matches_deployment@readonly [gw5] [ 46%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_gateway_matches_deployment@readonly test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_not_exposed_through_gateway@readonly [gw5] [ 47%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_not_exposed_through_gateway@readonly [gw0] [ 47%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestOrderingEdgeCases::test_subscription_before_auth_policy@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestManagedAnnotation::test_authpolicy_managed_false_prevents_update@api_keys [gw0] [ 48%] SKIPPED test/maas-e2e/test/e2e/tests/test_subscription.py::TestManagedAnnotation::test_authpolicy_managed_false_prevents_update@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestManagedAnnotation::test_trlp_managed_false_prevents_update@api_keys [gw2] [ 48%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_wrong_model_in_body_rejected@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_missing_model_in_body_rejected@tenant_isolation [gw0] [ 49%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestManagedAnnotation::test_trlp_managed_false_prevents_update@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_both_access_and_subscription_gets_200@api_keys [gw2] [ 49%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_missing_model_in_body_rejected@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_each_tenant_routes_to_own_model@tenant_isolation [gw0] [ 50%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_both_access_and_subscription_gets_200@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_subscription_but_no_access_gets_403@api_keys [gw0] [ 50%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_subscription_but_no_access_gets_403@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_multiple_subscriptions_separate_keys_gets_200@api_keys [gw0] [ 50%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_multiple_subscriptions_separate_keys_gets_200@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_mint_api_key_denied_for_inaccessible_subscription@api_keys [gw0] [ 51%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_mint_api_key_denied_for_inaccessible_subscription@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_access_gets_200@api_keys [gw0] [ 51%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_access_gets_200@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_subscription_but_no_auth_gets_403@api_keys [gw0] [ 52%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_subscription_but_no_auth_gets_403@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_active_status_with_valid_model@api_keys [gw2] [ 52%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_each_tenant_routes_to_own_model@tenant_isolation [gw1] [ 53%] FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_default_aitenant_bootstraps_maas_tenant_config_without_gateway_mutation@mt_lifecycle test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_rejected_outside_ai_tenants_namespace@mt_lifecycle [gw1] [ 53%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_rejected_outside_ai_tenants_namespace@mt_lifecycle test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_create_bootstrap_resources@mt_lifecycle [gw3] [ 54%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_explicit_subscription_header@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_empty_subscription_header_value@models [gw3] [ 54%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_empty_subscription_header_value@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_models_filtered_by_subscription@models [gw3] [ 55%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_models_filtered_by_subscription@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_deduplication_same_model_multiple_refs@models [gw3] [ 55%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_deduplication_same_model_multiple_refs@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_distinct_models_in_subscription@models [gw3] [ 56%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_distinct_models_in_subscription@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_user_token_returns_all_models@models [gw3] [ 56%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_user_token_returns_all_models@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_user_token_with_subscription_header_filters@models [gw3] [ 57%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_user_token_with_subscription_header_filters@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_response_schema_matches_openapi@models [gw3] [ 57%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_response_schema_matches_openapi@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_model_metadata_preserved@models [gw3] [ 58%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_model_metadata_preserved@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_scoped_to_subscription@models [gw3] [ 58%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_scoped_to_subscription@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_with_deleted_subscription_403@models [gw3] [ 59%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_with_deleted_subscription_403@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_with_inaccessible_subscription_403@models [gw3] [ 59%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_with_inaccessible_subscription_403@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_invalid_subscription_header_403@models [gw1] [ 60%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_create_bootstrap_resources@mt_lifecycle test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_migrates_and_removes_legacy_tenant@mt_lifecycle [gw3] [ 60%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_invalid_subscription_header_403@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_access_denied_to_subscription_403@models [gw3] [ 60%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_access_denied_to_subscription_403@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_ignores_subscription_header@models test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_creation_scoped_to_tenant@tenant_isolation [gw3] [ 61%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_ignores_subscription_header@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_api_keys_different_subscriptions@models [gw1] [ 61%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_migrates_and_removes_legacy_tenant@mt_lifecycle test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects@mt_lifecycle [gw3] [ 62%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_api_keys_different_subscriptions@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_service_account_token_multiple_subs_no_header@models [gw3] [ 62%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_service_account_token_multiple_subs_no_header@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_service_account_token_multiple_subs_with_header@models [gw0] [ 63%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_active_status_with_valid_model@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_failed_status_with_missing_model@api_keys [gw0] [ 63%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_failed_status_with_missing_model@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_active_status_with_valid_model@api_keys [gw0] [ 64%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_active_status_with_valid_model@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_failed_status_with_missing_model@api_keys [gw0] [ 64%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_failed_status_with_missing_model@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_degraded_status_with_partial_models@api_keys [gw1] [ 65%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects@mt_lifecycle test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_derives_non_default_tenant_namespace@mt_lifecycle [gw0] [ 65%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_degraded_status_with_partial_models@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_degraded_status_with_partial_models@api_keys [gw0] [ 66%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_degraded_status_with_partial_models@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_status_transitions_on_model_deletion@api_keys [gw3] [ 66%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_service_account_token_multiple_subs_with_header@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_unauthenticated_request_401@models [gw3] [ 67%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_unauthenticated_request_401@models test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyStructure::test_target_ref_points_to_gateway@models [gw3] [ 67%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyStructure::test_target_ref_points_to_gateway@models test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyStructure::test_no_per_model_authpolicy_for_fixture_model@models [gw3] [ 68%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyStructure::test_no_per_model_authpolicy_for_fixture_model@models test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyLifecycle::test_gateway_auth_rego_is_fixed_size@models [gw3] [ 68%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyLifecycle::test_gateway_auth_rego_is_fixed_size@models test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyLifecycle::test_only_one_gateway_authpolicy_named_maas_gateway_auth@models [gw3] [ 69%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyLifecycle::test_only_one_gateway_authpolicy_named_maas_gateway_auth@models test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_gateway_auth_group_membership_has_when_guard@models [gw3] [ 69%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_gateway_auth_group_membership_has_when_guard@models test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_gateway_auth_subscription_check_gated_by_model_identity@models [gw3] [ 70%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_gateway_auth_subscription_check_gated_by_model_identity@models test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_gateway_default_auth_scoped_if_present@models [gw3] [ 70%] SKIPPED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_gateway_default_auth_scoped_if_present@models [gw0] [ 70%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_status_transitions_on_model_deletion@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_degraded_healthy_model_allows_inference@api_keys [gw0] [ 71%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_degraded_healthy_model_allows_inference@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_failed_subscription_blocks_inference@api_keys [gw0] [ 71%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_failed_subscription_blocks_inference@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_models_endpoint_with_degraded_subscription_api_key@api_keys [gw0] [ 72%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_models_endpoint_with_degraded_subscription_api_key@api_keys test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_models_endpoint_with_degraded_subscription_kube_token@api_keys [gw0] [ 72%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_models_endpoint_with_degraded_subscription_kube_token@api_keys test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_returns_accessible_subscriptions@api_keys [gw0] [ 73%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_returns_accessible_subscriptions@api_keys test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_unauthenticated_returns_401@api_keys [gw0] [ 73%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_unauthenticated_returns_401@api_keys test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_subscription_includes_model_refs@api_keys [gw0] [ 74%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_subscription_includes_model_refs@api_keys test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_model_ref_display_name_and_description_enriched@api_keys [gw1] [ 74%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_derives_non_default_tenant_namespace@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_labeled_tenant_namespace_is_discovered@mt_lifecycle [gw1] [ 75%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_labeled_tenant_namespace_is_discovered@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_label_removal_stops_reconciliation@mt_lifecycle [gw0] [ 75%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_model_ref_display_name_and_description_enriched@api_keys test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_returns_subscriptions_for_model@api_keys [gw0] [ 76%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_returns_subscriptions_for_model@api_keys test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_unknown_model_returns_empty@api_keys [gw0] [ 76%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_unknown_model_returns_empty@api_keys test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_unauthenticated_returns_401@api_keys [gw0] [ 77%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_unauthenticated_returns_401@api_keys test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestSubscriptionModelAccessFiltering::test_filters_unauthorized_models@api_keys [gw0] [ 77%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestSubscriptionModelAccessFiltering::test_filters_unauthorized_models@api_keys test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestSubscriptionModelAccessFiltering::test_omits_subscription_with_no_authorized_models@api_keys [gw0] [ 78%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestSubscriptionModelAccessFiltering::test_omits_subscription_with_no_authorized_models@api_keys test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingPathRouting::test_embedding_path_based_200@api_keys [gw1] [ 78%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_label_removal_stops_reconciliation@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_unlabeled_namespace_ignored@mt_lifecycle [gw1] [ 79%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_unlabeled_namespace_ignored@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_dynamic_discovery_after_label_added@mt_lifecycle [gw1] [ 79%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_dynamic_discovery_after_label_added@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_per_tenant_oidc_configuration@mt_lifecycle [gw1] [ 80%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_per_tenant_oidc_configuration@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_namespace_qualified_collision_prevention@mt_lifecycle [gw2] [ 80%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_creation_scoped_to_tenant@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_validates_against_correct_tenant@tenant_isolation [gw1] [ 80%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_namespace_qualified_collision_prevention@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_tenant_admin_rbac_is_namespace_scoped@mt_lifecycle [gw0] [ 81%] PASSED test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingPathRouting::test_embedding_path_based_200@api_keys test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingPathRouting::test_embedding_bbr_llmisvc_200@api_keys [gw0] [ 81%] PASSED test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingPathRouting::test_embedding_bbr_llmisvc_200@api_keys [gw2] [ 82%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_validates_against_correct_tenant@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_rejected_cross_tenant@tenant_isolation [gw1] [ 82%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_tenant_admin_rbac_is_namespace_scoped@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantWebhookValidation::test_maassubscription_rejected_without_tenant_config_cr@mt_lifecycle [gw2] [ 83%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_rejected_cross_tenant@tenant_isolation [gw1] [ 83%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantWebhookValidation::test_maassubscription_rejected_without_tenant_config_cr@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantWebhookValidation::test_maasauthpolicy_rejected_without_tenant_config_cr@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_oidc_token_validation_per_tenant@tenant_isolation [gw2] [ 84%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_oidc_token_validation_per_tenant@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_list_scoped_to_tenant@tenant_isolation [gw1] [ 84%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantWebhookValidation::test_maasauthpolicy_rejected_without_tenant_config_cr@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantDiscoveryDormantMode::test_dormant_mode_ignores_labeled_namespace@mt_lifecycle [gw1] [ 85%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantDiscoveryDormantMode::test_dormant_mode_ignores_labeled_namespace@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestLegacyDefaultNamespaceStillWorks::test_models_as_a_service_namespace_reconciles@mt_lifecycle [gw1] [ 85%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestLegacyDefaultNamespaceStillWorks::test_models_as_a_service_namespace_reconciles@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_same_tenant_access@mt_lifecycle [gw2] [ 86%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_list_scoped_to_tenant@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_metadata_not_leaked_cross_tenant@tenant_isolation [gw2] [ 86%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_metadata_not_leaked_cross_tenant@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_subscription_selection_uses_tenant_namespace@tenant_isolation [gw2] [ 87%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_subscription_selection_uses_tenant_namespace@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_subscription_isolation.py::TestTenantSubscriptionIsolation::test_subscription_list_scoped_to_tenant@tenant_isolation [gw1] [ 87%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_same_tenant_access@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_cross_tenant_isolation@mt_lifecycle [gw1] [ 88%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_cross_tenant_isolation@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_unauthorized_access@mt_lifecycle [gw1] [ 88%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_unauthorized_access@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_each_tenant_returns_own_gateway@mt_lifecycle [gw1] [ 89%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_each_tenant_returns_own_gateway@mt_lifecycle test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_full_tenant_lifecycle_create_to_delete@mt_lifecycle [gw2] [ 89%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_subscription_isolation.py::TestTenantSubscriptionIsolation::test_subscription_list_scoped_to_tenant@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_subscription_isolation.py::TestTenantSubscriptionIsolation::test_subscription_selection_per_tenant@tenant_isolation [gw2] [ 90%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_subscription_isolation.py::TestTenantSubscriptionIsolation::test_subscription_selection_per_tenant@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_rate_limit_isolation.py::TestTenantRateLimitIsolation::test_rate_limit_enforced_per_tenant@tenant_isolation [gw1] [ 90%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_full_tenant_lifecycle_create_to_delete@mt_lifecycle test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_default_tenant_unaffected_by_multitenancy_enablement@mt_lifecycle [gw1] [ 90%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_default_tenant_unaffected_by_multitenancy_enablement@mt_lifecycle test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_same_named_resources_across_tenants@mt_lifecycle [gw1] [ 91%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_same_named_resources_across_tenants@mt_lifecycle test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_tenant_namespace_label_change_triggers_reconciliation@mt_lifecycle [gw2] [ 91%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_rate_limit_isolation.py::TestTenantRateLimitIsolation::test_rate_limit_enforced_per_tenant@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_rate_limit_isolation.py::TestTenantRateLimitIsolation::test_independent_tenant_rate_limits@tenant_isolation [gw2] [ 92%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_rate_limit_isolation.py::TestTenantRateLimitIsolation::test_independent_tenant_rate_limits@tenant_isolation test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_per_tenant_ipp_deployments_exist@tenant_isolation [gw1] [ 92%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_integration.py::TestMultiTenantIntegration::test_tenant_namespace_label_change_triggers_reconciliation@mt_lifecycle test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_aitenant_creates_dedicated_maas_api_infrastructure@mt_lifecycle [gw2] [ 93%] PASSED test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_per_tenant_ipp_deployments_exist@tenant_isolation test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_per_tenant_ipp_env_vars@tenant_isolation [gw2] [ 93%] PASSED test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_per_tenant_ipp_env_vars@tenant_isolation test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_per_tenant_envoyfilter_workload_selector_isolated@tenant_isolation [gw2] [ 94%] PASSED test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_per_tenant_envoyfilter_workload_selector_isolated@tenant_isolation test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_per_tenant_envoyfilter_grpc_clusters@tenant_isolation [gw2] [ 94%] PASSED test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_per_tenant_envoyfilter_grpc_clusters@tenant_isolation test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_default_tenant_keeps_legacy_ipp_names@tenant_isolation [gw2] [ 95%] PASSED test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_default_tenant_keeps_legacy_ipp_names@tenant_isolation test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_multiple_tenant_ipp_stacks_coexist@tenant_isolation [gw2] [ 95%] PASSED test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_multiple_tenant_ipp_stacks_coexist@tenant_isolation test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_per_tenant_networkpolicy_when_applied@tenant_isolation [gw2] [ 96%] PASSED test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPInfrastructure::test_per_tenant_networkpolicy_when_applied@tenant_isolation test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPRouting::test_default_gateway_hits_default_ipp_only@tenant_isolation [gw2] [ 96%] PASSED test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPRouting::test_default_gateway_hits_default_ipp_only@tenant_isolation test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPRouting::test_tenant_gateway_hits_tenant_ipp_only@tenant_isolation [gw1] [ 97%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_aitenant_creates_dedicated_maas_api_infrastructure@mt_lifecycle test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_tenant_name_environment_variable_set@mt_lifecycle [gw1] [ 97%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_tenant_name_environment_variable_set@mt_lifecycle test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_service_routing_isolation@mt_lifecycle [gw1] [ 98%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_service_routing_isolation@mt_lifecycle test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_httproute_tenant_attachment@mt_lifecycle [gw1] [ 98%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_httproute_tenant_attachment@mt_lifecycle test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_default_and_multiple_tenants_coexist@mt_lifecycle [gw1] [ 99%] PASSED test/maas-e2e/test/e2e/tests/test_multi_tenant_maas_api.py::TestPerTenantMaaSAPI::test_default_and_multiple_tenants_coexist@mt_lifecycle [gw2] [ 99%] PASSED test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPRouting::test_tenant_gateway_hits_tenant_ipp_only@tenant_isolation test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPCleanup::test_ipp_resources_removed_on_aitenant_delete@tenant_isolation [gw2] [100%] PASSED test/maas-e2e/test/e2e/tests/test_per_tenant_ipp_isolation.py::TestPerTenantIPPCleanup::test_ipp_resources_removed_on_aitenant_delete@tenant_isolation ==================================== ERRORS ==================================== _ ERROR at setup of TestTenantAuthIsolation.test_api_key_rejected_cross_tenant _ [gw2] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python self = def _new_conn(self) -> socket.socket: """Establish a socket connection and set nodelay settings on it. :return: New socket connection. """ try: > sock = connection.create_connection( (self._dns_host, self.port), self.timeout, source_address=self.source_address, socket_options=self.socket_options, ) test/e2e/.venv/lib64/python3.9/site-packages/urllib3/connection.py:204: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/urllib3/util/connection.py:60: in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ host = 'e2e-shared-a-w2-c0974e.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com' port = 443, family = type = , proto = 0, flags = 0 def getaddrinfo(host, port, family=0, type=0, proto=0, flags=0): """Resolve host and port into list of address info entries. Translate the host/port argument into a sequence of 5-tuples that contain all the necessary arguments for creating a socket connected to that service. host is a domain name, a string representation of an IPv4/v6 address or None. port is a string service name such as 'http', a numeric port number or None. By passing None as the value of host and port, you can pass NULL to the underlying C API. The family, type and proto arguments can be optionally specified in order to narrow the list of addresses returned. Passing zero as a value for each of these arguments selects the full range of results. """ # We override this function since we want to translate the numeric family # and socket type values to enum constants. addrlist = [] > for res in _socket.getaddrinfo(host, port, family, type, proto, flags): E socket.gaierror: [Errno -2] Name or service not known /usr/lib64/python3.9/socket.py:966: gaierror The above exception was the direct cause of the following exception: self = method = 'POST', url = '/maas-api/v1/api-keys' body = b'{"name": "e2e-auth-iso-a-74e726", "subscription": "e2e-auth-iso-fc1639"}' headers = {'User-Agent': 'python-requests/2.32.5', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-aliv...wT4X1nO2fIXEMes4BF0i0Cuj9NPNb-XbwNQCdYJ3seMF2TJKrd22hKc4w', 'Content-Type': 'application/json', 'Content-Length': '72'} retries = Retry(total=0, connect=None, read=False, redirect=None, status=None) redirect = False, assert_same_host = False timeout = Timeout(connect=45, read=45, total=None), pool_timeout = None release_conn = False, chunked = False, body_pos = None, preload_content = False decode_content = False, response_kw = {} parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/maas-api/v1/api-keys', query=None, fragment=None) destination_scheme = None, conn = None, release_this_conn = True http_tunnel_required = False, err = None, clean_exit = False def urlopen( # type: ignore[override] self, method: str, url: str, body: _TYPE_BODY | None = None, headers: typing.Mapping[str, str] | None = None, retries: Retry | bool | int | None = None, redirect: bool = True, assert_same_host: bool = True, timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, pool_timeout: int | None = None, release_conn: bool | None = None, chunked: bool = False, body_pos: _TYPE_BODY_POSITION | None = None, preload_content: bool = True, decode_content: bool = True, **response_kw: typing.Any, ) -> BaseHTTPResponse: """ Get a connection from the pool and perform an HTTP request. This is the lowest level call for making a request, so you'll need to specify all the raw details. .. note:: More commonly, it's appropriate to use a convenience method such as :meth:`request`. .. note:: `release_conn` will only behave as expected if `preload_content=False` because we want to make `preload_content=False` the default behaviour someday soon without breaking backwards compatibility. :param method: HTTP request method (such as GET, POST, PUT, etc.) :param url: The URL to perform the request on. :param body: Data to send in the request body, either :class:`str`, :class:`bytes`, an iterable of :class:`str`/:class:`bytes`, or a file-like object. :param headers: Dictionary of custom headers to send, such as User-Agent, If-None-Match, etc. If None, pool headers are used. If provided, these headers completely replace any pool-specific headers. :param retries: Configure the number of retries to allow before raising a :class:`~urllib3.exceptions.MaxRetryError` exception. If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a :class:`~urllib3.util.retry.Retry` object for fine-grained control over different types of retries. Pass an integer number to retry connection errors that many times, but no other types of errors. Pass zero to never retry. If ``False``, then retries are disabled and any exception is raised immediately. Also, instead of raising a MaxRetryError on redirects, the redirect response will be returned. :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. :param redirect: If True, automatically handle redirects (status codes 301, 302, 303, 307, 308). Each redirect counts as a retry. Disabling retries will disable redirect, too. :param assert_same_host: If ``True``, will make sure that the host of the pool requests is consistent else will raise HostChangedError. When ``False``, you can use the pool on an HTTP proxy and request foreign hosts. :param timeout: If specified, overrides the default timeout for this one request. It may be a float (in seconds) or an instance of :class:`urllib3.util.Timeout`. :param pool_timeout: If set and the pool is set to block=True, then this method will block for ``pool_timeout`` seconds and raise EmptyPoolError if no connection is available within the time period. :param bool preload_content: If True, the response's body will be preloaded into memory. :param bool decode_content: If True, will attempt to decode the body based on the 'content-encoding' header. :param release_conn: If False, then the urlopen call will not release the connection back into the pool once a response is received (but will release if you read the entire contents of the response such as when `preload_content=True`). This is useful if you're not preloading the response's content immediately. You will need to call ``r.release_conn()`` on the response ``r`` to return the connection back into the pool. If None, it takes the value of ``preload_content`` which defaults to ``True``. :param bool chunked: If True, urllib3 will send the body using chunked transfer encoding. Otherwise, urllib3 will send the body using the standard content-length form. Defaults to False. :param int body_pos: Position to seek to in file-like body in the event of a retry or redirect. Typically this won't need to be set because urllib3 will auto-populate the value when needed. """ parsed_url = parse_url(url) destination_scheme = parsed_url.scheme if headers is None: headers = self.headers if not isinstance(retries, Retry): retries = Retry.from_int(retries, redirect=redirect, default=self.retries) if release_conn is None: release_conn = preload_content # Check host if assert_same_host and not self.is_same_host(url): raise HostChangedError(self, url, retries) # Ensure that the URL we're connecting to is properly encoded if url.startswith("/"): url = to_str(_encode_target(url)) else: url = to_str(parsed_url.url) conn = None # Track whether `conn` needs to be released before # returning/raising/recursing. Update this variable if necessary, and # leave `release_conn` constant throughout the function. That way, if # the function recurses, the original value of `release_conn` will be # passed down into the recursive call, and its value will be respected. # # See issue #651 [1] for details. # # [1] release_this_conn = release_conn http_tunnel_required = connection_requires_http_tunnel( self.proxy, self.proxy_config, destination_scheme ) # Merge the proxy headers. Only done when not using HTTP CONNECT. We # have to copy the headers dict so we can safely change it without those # changes being reflected in anyone else's copy. if not http_tunnel_required: headers = headers.copy() # type: ignore[attr-defined] headers.update(self.proxy_headers) # type: ignore[union-attr] # Must keep the exception bound to a separate variable or else Python 3 # complains about UnboundLocalError. err = None # Keep track of whether we cleanly exited the except block. This # ensures we do proper cleanup in finally. clean_exit = False # Rewind body position, if needed. Record current position # for future rewinds in the event of a redirect/retry. body_pos = set_file_position(body, body_pos) try: # Request a connection from the queue. timeout_obj = self._get_timeout(timeout) conn = self._get_conn(timeout=pool_timeout) conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] # Is this a closed/new connection that requires CONNECT tunnelling? if self.proxy is not None and http_tunnel_required and conn.is_closed: try: self._prepare_proxy(conn) except (BaseSSLError, OSError, SocketTimeout) as e: self._raise_timeout( err=e, url=self.proxy.url, timeout_value=conn.timeout ) raise # If we're going to release the connection in ``finally:``, then # the response doesn't need to know about the connection. Otherwise # it will also try to release it and we'll have a double-release # mess. response_conn = conn if not release_conn else None # Make the request on the HTTPConnection object > response = self._make_request( conn, method, url, timeout=timeout_obj, body=body, headers=headers, chunked=chunked, retries=retries, response_conn=response_conn, preload_content=preload_content, decode_content=decode_content, **response_kw, ) test/e2e/.venv/lib64/python3.9/site-packages/urllib3/connectionpool.py:787: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/urllib3/connectionpool.py:488: in _make_request raise new_e test/e2e/.venv/lib64/python3.9/site-packages/urllib3/connectionpool.py:464: in _make_request self._validate_conn(conn) test/e2e/.venv/lib64/python3.9/site-packages/urllib3/connectionpool.py:1093: in _validate_conn conn.connect() test/e2e/.venv/lib64/python3.9/site-packages/urllib3/connection.py:759: in connect self.sock = sock = self._new_conn() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def _new_conn(self) -> socket.socket: """Establish a socket connection and set nodelay settings on it. :return: New socket connection. """ try: sock = connection.create_connection( (self._dns_host, self.port), self.timeout, source_address=self.source_address, socket_options=self.socket_options, ) except socket.gaierror as e: > raise NameResolutionError(self.host, self, e) from e E urllib3.exceptions.NameResolutionError: HTTPSConnection(host='e2e-shared-a-w2-c0974e.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com', port=443): Failed to resolve 'e2e-shared-a-w2-c0974e.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com' ([Errno -2] Name or service not known) test/e2e/.venv/lib64/python3.9/site-packages/urllib3/connection.py:211: NameResolutionError The above exception was the direct cause of the following exception: self = request = , stream = False timeout = Timeout(connect=45, read=45, total=None), verify = False, cert = None proxies = OrderedDict() def send( self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None ): """Sends PreparedRequest object. Returns Response object. :param request: The :class:`PreparedRequest ` being sent. :param stream: (optional) Whether to stream the request content. :param timeout: (optional) How long to wait for the server to send data before giving up, as a float, or a :ref:`(connect timeout, read timeout) ` tuple. :type timeout: float or tuple or urllib3 Timeout object :param verify: (optional) Either a boolean, in which case it controls whether we verify the server's TLS certificate, or a string, in which case it must be a path to a CA bundle to use :param cert: (optional) Any user-provided SSL certificate to be trusted. :param proxies: (optional) The proxies dictionary to apply to the request. :rtype: requests.Response """ try: conn = self.get_connection_with_tls_context( request, verify, proxies=proxies, cert=cert ) except LocationValueError as e: raise InvalidURL(e, request=request) self.cert_verify(conn, request.url, verify, cert) url = self.request_url(request, proxies) self.add_headers( request, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies, ) chunked = not (request.body is None or "Content-Length" in request.headers) if isinstance(timeout, tuple): try: connect, read = timeout timeout = TimeoutSauce(connect=connect, read=read) except ValueError: raise ValueError( f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, " f"or a single float to set both timeouts to the same value." ) elif isinstance(timeout, TimeoutSauce): pass else: timeout = TimeoutSauce(connect=timeout, read=timeout) try: > resp = conn.urlopen( method=request.method, url=url, body=request.body, headers=request.headers, redirect=False, assert_same_host=False, preload_content=False, decode_content=False, retries=self.max_retries, timeout=timeout, chunked=chunked, ) test/e2e/.venv/lib64/python3.9/site-packages/requests/adapters.py:644: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/urllib3/connectionpool.py:841: in urlopen retries = retries.increment( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = Retry(total=0, connect=None, read=False, redirect=None, status=None) method = 'POST', url = '/maas-api/v1/api-keys', response = None error = NameResolutionError("HTTPSConnection(host='e2e-shared-a-w2-c0974e.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfl...ed-a-w2-c0974e.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com' ([Errno -2] Name or service not known)") _pool = _stacktrace = def increment( self, method: str | None = None, url: str | None = None, response: BaseHTTPResponse | None = None, error: Exception | None = None, _pool: ConnectionPool | None = None, _stacktrace: TracebackType | None = None, ) -> Self: """Return a new Retry object with incremented retry counters. :param response: A response object, or None, if the server did not return a response. :type response: :class:`~urllib3.response.BaseHTTPResponse` :param Exception error: An error encountered during the request, or None if the response was received successfully. :return: A new ``Retry`` object. """ if self.total is False and error: # Disabled, indicate to re-raise the error. raise reraise(type(error), error, _stacktrace) total = self.total if total is not None: total -= 1 connect = self.connect read = self.read redirect = self.redirect status_count = self.status other = self.other cause = "unknown" status = None redirect_location = None if error and self._is_connection_error(error): # Connect retry? if connect is False: raise reraise(type(error), error, _stacktrace) elif connect is not None: connect -= 1 elif error and self._is_read_error(error): # Read retry? if read is False or method is None or not self._is_method_retryable(method): raise reraise(type(error), error, _stacktrace) elif read is not None: read -= 1 elif error: # Other retry? if other is not None: other -= 1 elif response and response.get_redirect_location(): # Redirect retry? if redirect is not None: redirect -= 1 cause = "too many redirects" response_redirect_location = response.get_redirect_location() if response_redirect_location: redirect_location = response_redirect_location status = response.status else: # Incrementing because of a server error like a 500 in # status_forcelist and the given method is in the allowed_methods cause = ResponseError.GENERIC_ERROR if response and response.status: if status_count is not None: status_count -= 1 cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status) status = response.status history = self.history + ( RequestHistory(method, url, error, status, redirect_location), ) new_retry = self.new( total=total, connect=connect, read=read, redirect=redirect, status=status_count, other=other, history=history, ) if new_retry.is_exhausted(): reason = error or ResponseError(cause) > raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] E urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='e2e-shared-a-w2-c0974e.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com', port=443): Max retries exceeded with url: /maas-api/v1/api-keys (Caused by NameResolutionError("HTTPSConnection(host='e2e-shared-a-w2-c0974e.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com', port=443): Failed to resolve 'e2e-shared-a-w2-c0974e.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com' ([Errno -2] Name or service not known)")) test/e2e/.venv/lib64/python3.9/site-packages/urllib3/util/retry.py:535: MaxRetryError During handling of the above exception, another exception occurred: tenant_auth_setup = {'policy': 'e2e-auth-iso-fc1639', 'subscription': 'e2e-auth-iso-fc1639', 'tenant_a': {'base_url': 'https://e2e-shared-...ared-b-w2-d31d7c', 'model_name': 'auth-test-model-d31d7c', 'model_namespace': 'ai-tenant-e2e-shared-b-w2-d31d7c', ...}} @pytest.fixture def tenant_api_keys(tenant_auth_setup): oc_token = _get_cluster_token() created = {} for key_name, tenant in (("a", tenant_auth_setup["tenant_a"]), ("b", tenant_auth_setup["tenant_b"])): > response = create_api_key_at( tenant["base_url"], oc_token, f"e2e-auth-iso-{key_name}-{uuid.uuid4().hex[:6]}", subscription=tenant_auth_setup["subscription"], ) test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py:159: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:1397: in create_api_key_at return _request_with_gateway_retry( test/maas-e2e/test/e2e/tests/test_helper.py:518: in _request_with_gateway_retry r = method(url, timeout=timeout, verify=verify, **kwargs) test/e2e/.venv/lib64/python3.9/site-packages/requests/api.py:115: in post return request("post", url, data=data, json=json, **kwargs) test/e2e/.venv/lib64/python3.9/site-packages/requests/api.py:59: in request return session.request(method=method, url=url, **kwargs) test/e2e/.venv/lib64/python3.9/site-packages/requests/sessions.py:589: in request resp = self.send(prep, **send_kwargs) test/e2e/.venv/lib64/python3.9/site-packages/requests/sessions.py:703: in send r = adapter.send(request, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = request = , stream = False timeout = Timeout(connect=45, read=45, total=None), verify = False, cert = None proxies = OrderedDict() def send( self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None ): """Sends PreparedRequest object. Returns Response object. :param request: The :class:`PreparedRequest ` being sent. :param stream: (optional) Whether to stream the request content. :param timeout: (optional) How long to wait for the server to send data before giving up, as a float, or a :ref:`(connect timeout, read timeout) ` tuple. :type timeout: float or tuple or urllib3 Timeout object :param verify: (optional) Either a boolean, in which case it controls whether we verify the server's TLS certificate, or a string, in which case it must be a path to a CA bundle to use :param cert: (optional) Any user-provided SSL certificate to be trusted. :param proxies: (optional) The proxies dictionary to apply to the request. :rtype: requests.Response """ try: conn = self.get_connection_with_tls_context( request, verify, proxies=proxies, cert=cert ) except LocationValueError as e: raise InvalidURL(e, request=request) self.cert_verify(conn, request.url, verify, cert) url = self.request_url(request, proxies) self.add_headers( request, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies, ) chunked = not (request.body is None or "Content-Length" in request.headers) if isinstance(timeout, tuple): try: connect, read = timeout timeout = TimeoutSauce(connect=connect, read=read) except ValueError: raise ValueError( f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, " f"or a single float to set both timeouts to the same value." ) elif isinstance(timeout, TimeoutSauce): pass else: timeout = TimeoutSauce(connect=timeout, read=timeout) try: resp = conn.urlopen( method=request.method, url=url, body=request.body, headers=request.headers, redirect=False, assert_same_host=False, preload_content=False, decode_content=False, retries=self.max_retries, timeout=timeout, chunked=chunked, ) except (ProtocolError, OSError) as err: raise ConnectionError(err, request=request) except MaxRetryError as e: if isinstance(e.reason, ConnectTimeoutError): # TODO: Remove this in 3.0.0: see #2811 if not isinstance(e.reason, NewConnectionError): raise ConnectTimeout(e, request=request) if isinstance(e.reason, ResponseError): raise RetryError(e, request=request) if isinstance(e.reason, _ProxyError): raise ProxyError(e, request=request) if isinstance(e.reason, _SSLError): # This branch is for urllib3 v1.22 and later. raise SSLError(e, request=request) > raise ConnectionError(e, request=request) E requests.exceptions.ConnectionError: HTTPSConnectionPool(host='e2e-shared-a-w2-c0974e.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com', port=443): Max retries exceeded with url: /maas-api/v1/api-keys (Caused by NameResolutionError("HTTPSConnection(host='e2e-shared-a-w2-c0974e.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com', port=443): Failed to resolve 'e2e-shared-a-w2-c0974e.apps.5fca5911-ab5d-4a6b-b5ce-4b82eaa71ecb.prod.konfluxeaas.com' ([Errno -2] Name or service not known)")) test/e2e/.venv/lib64/python3.9/site-packages/requests/adapters.py:677: ConnectionError ------------------------------ Captured log setup ------------------------------ INFO test_helper:test_helper.py:1381 Waiting for MaaSSubscription e2e-auth-iso-fc1639 TRLP ready=True (timeout: 120s)... INFO test_helper:test_helper.py:1399 ✅ MaaSSubscription e2e-auth-iso-fc1639 has 1 TRLP(s) with ready=True INFO test_helper:test_helper.py:1381 Waiting for MaaSSubscription e2e-auth-iso-fc1639 TRLP ready=True (timeout: 120s)... INFO test_helper:test_helper.py:1399 ✅ MaaSSubscription e2e-auth-iso-fc1639 has 1 TRLP(s) with ready=True INFO test_helper:test_helper.py:479 Using TOKEN env var for API key operations =================================== FAILURES =================================== ___________ TestTenantLifecycle.test_tenant_ready_and_phase_healthy ____________ [gw5] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python self = def test_tenant_ready_and_phase_healthy(self): st = _wait_tenant_ready() > assert st is not None, "MaasTenantConfig Ready did not become True in time." E AssertionError: MaasTenantConfig Ready did not become True in time. E assert None is not None test/maas-e2e/test/e2e/tests/test_tenant.py:210: AssertionError _ TestAITenantLifecycle.test_default_aitenant_bootstraps_maas_tenant_config_without_gateway_mutation _ [gw1] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python self = def test_default_aitenant_bootstraps_maas_tenant_config_without_gateway_mutation(self): > aitenant = _wait_for_json( AITENANT_KIND, DEFAULT_AITENANT_NAME, AITENANT_NAMESPACE, predicate=_aitenant_ready, timeout=240, ) test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:296: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'models-as-a-service', namespace = 'ai-tenants' def _wait_for_json(kind, name, namespace=None, *, predicate=None, timeout=180, interval=5): deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = _get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError(f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}") E AssertionError: aitenant/models-as-a-service in ai-tenants did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'AITenant', 'metadata': {'annotations': {'maas.opendatahub.io/payload-processing-type': 'praxis'}, 'creationTimestamp': '2026-09-16T14:18:12Z', 'finalizers': ['maas.opendatahub.io/aitenant-cleanup', 'ai-gateway-controller.opendatahub.io/praxis-cleanup'], 'generation': 1, 'name': 'models-as-a-service', 'namespace': 'ai-tenants', 'ownerReferences': [{'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'Config', 'name': 'default', 'uid': '187d6fa2-fd3b-48f0-a39a-98b5703c43d8'}], 'resourceVersion': '25612', 'uid': '261bb904-74d0-4f49-9f10-aebd6ab10825'}, 'spec': {'gateway': {'name': 'maas-default-gateway'}}, 'status': {'conditions': [{'lastTransitionTime': '2026-09-16T14:21:24Z', 'message': 'waiting for MaasTenantConfig to report Ready', 'observedGeneration': 1, 'reason': 'TenantConfigNotReady', 'status': 'False', 'type': 'Ready'}], 'gatewayRef': {'name': 'maas-default-gateway', 'namespace': 'openshift-ingress'}, 'phase': 'Pending', 'tenantNamespace': 'models-as-a-service'}} test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:113: AssertionError ------ generated xml file: /workspace/artifacts-dir/e2e-system:admin.xml ------- - Generated html report: file:///workspace/artifacts-dir/e2e-system%3Aadmin.html - =========================== short test summary info ============================ FAILED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_tenant_ready_and_phase_healthy@readonly FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_default_aitenant_bootstraps_maas_tenant_config_without_gateway_mutation@mt_lifecycle ERROR test/maas-e2e/test/e2e/tests/test_tenant_auth_isolation.py::TestTenantAuthIsolation::test_api_key_rejected_cross_tenant@tenant_isolation = 2 failed, 190 passed, 17 skipped, 150 warnings, 1 error in 1240.70s (0:20:40) = Running E2E pass 2/2: serial cluster mutators (-m serial, single worker) ============================= test session starts ============================== platform linux -- Python 3.9.25, pytest-8.4.2, pluggy-1.6.0 -- /workspace/source/test/e2e/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.9.25', 'Platform': 'Linux-5.14.0-570.134.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '8.4.2', 'pluggy': '1.6.0'}, 'Plugins': {'xdist': '3.8.0', 'metadata': '3.1.1', 'html': '4.2.0'}, 'PLATFORM': 'el9'} rootdir: /workspace/source/test/maas-e2e/test/e2e configfile: pyproject.toml plugins: xdist-3.8.0, metadata-3.1.1, html-4.2.0 collecting ... collected 233 items / 210 deselected / 23 selected test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_pending_subscription [token] using env TOKEN (masked): 1016 PASSED [ 4%] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_reject_key_for_unreconciled_subscription PASSED [ 8%] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAuthPolicyRemoval::test_authpolicy_deletion_revokes_access PASSED [ 13%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_rate_limit_exhaustion_gets_429 PASSED [ 17%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_models_endpoint_exempt_from_rate_limiting PASSED [ 21%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestMultipleSubscriptionsPerModel::test_user_in_one_of_two_subscriptions_gets_200 PASSED [ 26%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestMultipleAuthPoliciesPerModel::test_delete_one_auth_policy_other_still_works PASSED [ 30%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestCascadeDeletion::test_delete_subscription_rebuilds_trlp PASSED [ 34%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestCascadeDeletion::test_trlp_persists_during_multi_subscription_deletion PASSED [ 39%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestCascadeDeletion::test_delete_last_subscription_denies_access PASSED [ 43%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestCascadeDeletion::test_unconfigured_model_denied_by_gateway_auth PASSED [ 47%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_access_but_no_subscription_gets_403 PASSED [ 52%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_single_subscription_auto_selects PASSED [ 56%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_auth_but_no_subscription_gets_403 PASSED [ 60%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_degraded_trlp_blocks_inference PASSED [ 65%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_single_subscription_auto_select PASSED [ 69%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_different_modelrefs_same_model_id maasauthpolicy.maas.opendatahub.io/e2e-diff-refs-auth created maassubscription.maas.opendatahub.io/e2e-diff-refs-subscription created PASSED [ 73%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_empty_model_list PASSED [ 78%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_central_models_endpoint_exempt_from_rate_limiting PASSED [ 82%] test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_payload_processing_deployed_with_active_tenant FAILED [ 86%] test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingGovernance::test_embedding_default_deny_403 PASSED [ 91%] test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingGovernance::test_embedding_trlp_429 PASSED [ 95%] test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingGovernance::test_embedding_with_governance_200 PASSED [100%] =================================== FAILURES =================================== ___ TestTenantLifecycle.test_payload_processing_deployed_with_active_tenant ____ self = @pytest.mark.serial def test_payload_processing_deployed_with_active_tenant(self): """Active MaasTenantConfig should reconcile tenant platform workloads. Verifies payload-processing exists and that spec.maasApi replicas/resources overrides are applied to the default maas-api Deployment (then restored). """ st = _wait_tenant_ready() > assert st is not None, "MaasTenantConfig not Ready; skip workload checks." E AssertionError: MaasTenantConfig not Ready; skip workload checks. E assert None is not None test/maas-e2e/test/e2e/tests/test_tenant.py:225: AssertionError --- generated xml file: /workspace/artifacts-dir/e2e-system:admin-serial.xml --- - Generated html report: file:///workspace/artifacts-dir/e2e-system%3Aadmin-serial.html - =========================== short test summary info ============================ FAILED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_payload_processing_deployed_with_active_tenant ==== 1 failed, 22 passed, 210 deselected, 27 warnings in 585.53s (0:09:45) ===== ERROR: E2E tests failed (parallel_rc=1, serial_rc=1)