Fetching https://github.com/opendatahub-io/models-as-a-service@7879675aa66d002425c7b32375d6d0156eaa0c56 into /workspace/source/test/maas-e2e ... MaaS checkout at: 7879675aa66d002425c7b32375d6d0156eaa0c56 Patched /workspace/source/test/maas-e2e/scripts/deploy.sh Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS checkout fixture root) NOTE: rewriting AI_GATEWAY_CONTROLLER_IMAGE from snapshot digest to tag odh-pr-146 MaaS platform images: MAAS_API_IMAGE=quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 MAAS_CONTROLLER_IMAGE=quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 AI_GATEWAY_CONTROLLER_IMAGE=quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-146 PRAXIS_EXTPROC_IMAGE=quay.io/opendatahub/odh-praxis-extproc:odh-stable ---------------------------------------- ai-gateway-controller E2E on OpenShift ---------------------------------------- Checking prerequisites... Prerequisites met — logged in as: system:admin DEPLOY_MODE: kustomize MAAS_API_IMAGE: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 MAAS_CONTROLLER_IMAGE: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 AI_GATEWAY_CONTROLLER_IMAGE: quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-146 PRAXIS_EXTPROC_IMAGE: quay.io/opendatahub/odh-praxis-extproc:odh-stable 2026-11-02T20:19:22Z deploy_platform start Deploying MaaS platform via ODH operator... Gateway ingress mode for deploy.sh: ocproute (loadbalancer | ocproute) Using custom MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 Using custom MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 Deployment mode: kustomize Using policy engine: rhcl (Authorino namespace: kuadrant-system) [INFO] =================================================== [INFO] Models-as-a-Service Deployment [INFO] =================================================== [INFO] Validating configuration... [INFO] Configuration validated successfully [INFO] Deployment configuration: [INFO] Mode: kustomize [INFO] Operator: odh [INFO] Policy Engine: rhcl [INFO] Namespace: opendatahub [INFO] TLS Backend: true [INFO] External OIDC: false [INFO] MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] Installing shared dependencies for kustomize mode... [INFO] =================================================== [INFO] Setup Shared Dependencies (Helm) [INFO] =================================================== [INFO] Operator type: odh [INFO] Deployment mode: kustomize [INFO] Ingress mode: ocproute [INFO] Policy engine: rhcl [INFO] Chart source: https://github.com/opendatahub-io/odh-gitops.git @ main (fadfb31e89ae898b04a9a75bfa8bb91588d7e625) [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] Payload processing image: chart default [INFO] AI Gateway operator image: catalog default [INFO] helm not found, installing v3.17.3... [INFO] helm installed: v3.17.3+ge4da497 [INFO] Cloning odh-gitops chart (branch: main, commit: fadfb31e89ae898b04a9a75bfa8bb91588d7e625)... Cloning into '/tmp/tmp.DRRp4ZL2C9/odh-gitops'... From https://github.com/opendatahub-io/odh-gitops * branch fadfb31e89ae898b04a9a75bfa8bb91588d7e625 -> FETCH_HEAD [INFO] Chart resolved at: /tmp/tmp.DRRp4ZL2C9/odh-gitops/charts/rhai-on-openshift-chart [INFO] [INFO] Phase 1: Installing operators via Helm chart... [INFO] Running: helm upgrade --install rhoai-deps Release "rhoai-deps" does not exist. Installing it now. NAME: rhoai-deps LAST DEPLOYED: Mon Nov 2 20:19:24 2026 NAMESPACE: rhoai-deps STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: ODH Chart installed successfully! === Installation Summary === Operator: ODH Namespace: rhoai-deps Profile: maas === Enabled Components === - aigateway: Removed - aipipelines: Removed - dashboard: Removed - feastoperator: Removed - kserve: Managed - kueue: Removed - mlflowoperator: Removed - modelregistry: Removed - ogx: Removed - ray: Removed - sparkoperator: Removed - trainer: Removed - trainingoperator: Removed - trustyai: Removed - workbenches: Removed === Enabled Services === - monitoring: Removed === Enabled Dependencies === - certManager - jobSet - rhcl === Next Steps === 1. Wait for operators to be ready: kubectl get csv -A | grep -E "(rhods|opendatahub|cert-manager|leader-worker)" 2. If CRs were not created (first install), run the same helm upgrade command again. 3. Check DataScienceCluster status: kubectl get datasciencecluster default-dsc -o jsonpath='{.status.phase}' === Enable Authorino TLS (Optional) === To enable TLS for Authorino: 1. Annotate the service to trigger TLS certificate generation: kubectl annotate svc/authorino-authorino-authorization \ service.beta.openshift.io/serving-cert-secret-name=authorino-server-cert \ -n kuadrant-system 2. Patch the Authorino CR to enable TLS: kubectl patch authorino/authorino -n kuadrant-system --type=merge -p '{"spec":{"clusterWide":true,"listener":{"tls":{"enabled":true,"certSecretRef":{"name":"authorino-server-cert"}}},"oidcServer":{"tls":{"enabled":false}}}}' === Documentation === For more information, see the chart README: https://github.com/opendatahub-io/odh-gitops/tree/main/charts/rhai-on-openshift-chart [INFO] Enabling bundle unpack retry on OperatorGroup cert-manager-operator/openshift-cert-manager-operator operatorgroup.operators.coreos.com/openshift-cert-manager-operator annotated [INFO] Enabling bundle unpack retry on OperatorGroup kuadrant-system/rhcl-operator operatorgroup.operators.coreos.com/rhcl-operator annotated [INFO] Enabling bundle unpack retry on OperatorGroup opendatahub-operator-system/opendatahub-operator operatorgroup.operators.coreos.com/opendatahub-operator annotated [INFO] Enabling bundle unpack retry on OperatorGroup openshift-jobset-operator/job-set operatorgroup.operators.coreos.com/job-set annotated [INFO] [INFO] Waiting for OLM to unpack operator bundles... [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator opendatahub-operator-system/opendatahub-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator opendatahub-operator-system/opendatahub-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator opendatahub-operator-system/opendatahub-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator opendatahub-operator-system/opendatahub-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator opendatahub-operator-system/opendatahub-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator (2000s timeout) [INFO] Operator bundles unpacked [INFO] [INFO] Waiting for operator CRDs to be established... ⏳ Waiting for CRD dscinitializations.dscinitialization.opendatahub.io to appear (timeout: 600s)… ✅ CRD dscinitializations.dscinitialization.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io condition met ⏳ Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 600s)… ✅ CRD datascienceclusters.datasciencecluster.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met ⏳ Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 600s)… ✅ CRD kuadrants.kuadrant.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [INFO] [INFO] Waiting for ODH operator to be ready (webhook must be serving)... [INFO] Operator CSV phase: Installing (0s / 300s) [INFO] Operator CSV is Succeeded in opendatahub-operator-system deployment.apps/opendatahub-operator-controller-manager condition met [INFO] [INFO] Applying local MaaS CRDs for kustomize mode... ⏳ Applying MaaS Controller CRDs from /workspace/source/test/maas-e2e/deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io created ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] [INFO] Phase 2: Applying CRD-dependent resources (DSC, DSCI, Kuadrant CR)... [INFO] Running: helm upgrade --install rhoai-deps Release "rhoai-deps" has been upgraded. Happy Helming! NAME: rhoai-deps LAST DEPLOYED: Mon Nov 2 20:20:48 2026 NAMESPACE: rhoai-deps STATUS: deployed REVISION: 2 TEST SUITE: None NOTES: ODH Chart installed successfully! === Installation Summary === Operator: ODH Namespace: rhoai-deps Profile: maas === Enabled Components === - aigateway: Removed - aipipelines: Removed - dashboard: Removed - feastoperator: Removed - kserve: Managed - kueue: Removed - mlflowoperator: Removed - modelregistry: Removed - ogx: Removed - ray: Removed - sparkoperator: Removed - trainer: Removed - trainingoperator: Removed - trustyai: Removed - workbenches: Removed === Enabled Services === - monitoring: Removed === Enabled Dependencies === - certManager - jobSet - rhcl === Next Steps === 1. Wait for operators to be ready: kubectl get csv -A | grep -E "(rhods|opendatahub|cert-manager|leader-worker)" 2. If CRs were not created (first install), run the same helm upgrade command again. 3. Check DataScienceCluster status: kubectl get datasciencecluster default-dsc -o jsonpath='{.status.phase}' === Enable Authorino TLS (Optional) === To enable TLS for Authorino: 1. Annotate the service to trigger TLS certificate generation: kubectl annotate svc/authorino-authorino-authorization \ service.beta.openshift.io/serving-cert-secret-name=authorino-server-cert \ -n kuadrant-system 2. Patch the Authorino CR to enable TLS: kubectl patch authorino/authorino -n kuadrant-system --type=merge -p '{"spec":{"clusterWide":true,"listener":{"tls":{"enabled":true,"certSecretRef":{"name":"authorino-server-cert"}}},"oidcServer":{"tls":{"enabled":false}}}}' === Documentation === For more information, see the chart README: https://github.com/opendatahub-io/odh-gitops/tree/main/charts/rhai-on-openshift-chart [INFO] Waiting for Kuadrant policy engine to become ready... [INFO] Waiting for: Kuadrant ready in kuadrant-system (timeout: 600s) [INFO] Kuadrant ready in kuadrant-system - Ready [INFO] [INFO] Shared dependencies installed successfully [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [WARN] DEPLOYING POC POSTGRESQL — NOT INTENDED FOR PRODUCTION USE [WARN] Data is stored in ephemeral storage and will be lost on pod restart. [WARN] For production, use --postgres-connection with an external database [WARN] (AWS RDS, Crunchy Operator, Azure Database, etc.) [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ⚠️ WARNING FOR PRODUCTION USE. ⚠️ ┃ ┃ This deploys PostgreSQL with ephemeral storage (emptyDir). ┃ ┃ Data WILL be lost on pod restart. ┃ ┃ For production, use an external database: ┃ ┃ deploy.sh --postgres-connection postgresql://... ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 🔧 Fresh install: Deploying PostgreSQL in infrastructure namespace 'odh-ai-gateway-infra'... 📦 Creating infrastructure namespace 'odh-ai-gateway-infra'... namespace/odh-ai-gateway-infra created Generated random PostgreSQL password (stored in secret postgres-creds) Creating PostgreSQL deployment... ⚠️ Using POC configuration (ephemeral storage) Using default PostgreSQL image (operator CSV not available) Image: registry.redhat.io/rhel9/postgresql-16:latest secret/postgres-creds created deployment.apps/postgres created service/postgres created secret/maas-db-config created Waiting for PostgreSQL to be ready... deployment.apps/postgres condition met ✅ PostgreSQL deployed successfully Namespace: odh-ai-gateway-infra Database: maas User: maas Secret: maas-db-config (contains DB_CONNECTION_URL with FQDN) ⚠️ For production, use AWS RDS, Crunchy Operator, or Azure Database Note: Schema migrations run automatically when maas-api starts [INFO] Shared dependencies installed [INFO] [INFO] MaaS Controller... [INFO] Phase 1: Applying MaaS CRDs and waiting until Established (controller creates Config after CRD is ready)... ⏳ Applying MaaS Controller CRDs from /workspace/source/test/maas-e2e/scripts/../deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io unchanged ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] Phase 2: Applying full controller kustomize (same as operator: deployment/base/maas-controller/default)... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io configured serviceaccount/maas-controller created role.rbac.authorization.k8s.io/maas-controller-leader-election-role created clusterrole.rbac.authorization.k8s.io/maas-controller-cluster-config-role created clusterrole.rbac.authorization.k8s.io/maas-controller-ocp-role created clusterrole.rbac.authorization.k8s.io/maas-controller-role created clusterrole.rbac.authorization.k8s.io/maas-owner-role created clusterrole.rbac.authorization.k8s.io/maas-viewer-role created rolebinding.rbac.authorization.k8s.io/maas-controller-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-cluster-config-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-ocp-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-rolebinding created configmap/maas-parameters created service/maas-controller-metrics created service/maas-controller-webhook-service created deployment.apps/maas-controller created servicemonitor.monitoring.coreos.com/maas-controller-metrics created networkpolicy.networking.k8s.io/maas-controller-allow-monitoring created validatingwebhookconfiguration.admissionregistration.k8s.io/maas-validating-webhook-configuration created [INFO] Restarting maas-controller to pick up configuration changes... deployment.apps/maas-controller restarted [INFO] Waiting for maas-controller to be ready... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... deployment "maas-controller" successfully rolled out [INFO] Controller ready. [INFO] [INFO] Waiting for Tenant reconciler to deploy maas-api... [INFO] Applying secret migration RBAC to namespace odh-ai-gateway-infra... role.rbac.authorization.k8s.io/maas-controller-secret-migrate created rolebinding.rbac.authorization.k8s.io/maas-controller-secret-migrate created [INFO] maas-api deployment found in odh-ai-gateway-infra, waiting for rollout... Waiting for deployment "maas-api" rollout to finish: 0 of 1 updated replicas are available... deployment "maas-api" successfully rolled out [INFO] maas-api is ready [INFO] Configuring TLS backend for Authorino and MaaS API... * Waiting for deployment/authorino in kuadrant-system (timeout: 300s)... * Found deployment/authorino [INFO] Running TLS configuration script... [INFO] TLS configuration script completed successfully [INFO] Restarting deployments to pick up TLS configuration... deployment.apps/maas-api restarted deployment.apps/authorino restarted [INFO] Waiting for Authorino deployment to be ready... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment spec update to be observed... Waiting for deployment spec update to be observed... Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out [INFO] TLS backend configuration complete [INFO] [INFO] MaaS API and MaaS Controller deployment completed successfully! [INFO] maas-api image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 (namespace: odh-ai-gateway-infra) [INFO] maas-controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 (namespace: opendatahub) [INFO] =================================================== [INFO] Models-as-a-Service Deployment completed successfully! [INFO] =================================================== * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: (informational only), AIGatewayReady: False - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: (informational only), AIGatewayReady: False - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: (informational only), AIGatewayReady: False * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' Raising llmisvc-controller-manager CPU in opendatahub (request 1, limit 2)... deployment.apps/llmisvc-controller-manager resource requirements updated Waiting for deployment "llmisvc-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "llmisvc-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "llmisvc-controller-manager" rollout to finish: 1 old replicas are pending termination... deployment "llmisvc-controller-manager" successfully rolled out llmisvc-controller-manager resources: {"limits":{"cpu":"2","memory":"300Mi"},"requests":{"cpu":"1","memory":"300Mi"}} ⚠️ WARNING: Skipping Authorino readiness check (SKIP_AUTH_CHECK=true) ✅ MaaS platform deployment completed 2026-11-02T20:24:46Z deploy_platform end ---------------------------------------- Deploying Models ---------------------------------------- 2026-11-02T20:24:46Z deploy_models start Deploying MaaS system (free + premium: LLMIS + MaaSModelRef + MaaSAuthPolicy + MaaSSubscription) Waiting for Gateway openshift-ingress/maas-default-gateway to be Programmed=True (timeout: 600s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met ✅ Gateway openshift-ingress/maas-default-gateway is Programmed Creating 'llm' namespace... namespace/llm created 'models-as-a-service' namespace already exists Waiting for ready endpoints on maas-controller-webhook-service in opendatahub (timeout: 120s)... ✅ MaaS admission webhook has ready endpoints: 10.134.0.27 maasauthpolicy.maas.opendatahub.io/premium-simulator-access created maasauthpolicy.maas.opendatahub.io/simulator-access created maasmodelref.maas.opendatahub.io/e2e-distinct-2-simulated created maasmodelref.maas.opendatahub.io/e2e-distinct-simulated created maasmodelref.maas.opendatahub.io/e2e-embedding-simulated created maasmodelref.maas.opendatahub.io/e2e-trlp-test-simulated created maasmodelref.maas.opendatahub.io/e2e-unconfigured-facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/echo-simulator created maasmodelref.maas.opendatahub.io/facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/premium-simulated-simulated-premium created maassubscription.maas.opendatahub.io/premium-simulator-subscription created maassubscription.maas.opendatahub.io/simulator-subscription created llminferenceservice.serving.kserve.io/e2e-distinct-2-simulated created llminferenceservice.serving.kserve.io/e2e-distinct-simulated created llminferenceservice.serving.kserve.io/e2e-embedding-simulated created llminferenceservice.serving.kserve.io/e2e-trlp-test-simulated created llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/echo-simulator created llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium created ✅ MaaS system deployed (free + premium + e2e test fixtures) Waiting for models to be ready (timeout: 300s)... llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated condition met llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium condition met llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated condition met ✅ Simulator models ready Waiting for governed MaaSModelRefs to be Ready (timeout: 300s per model)... maasmodelref.maas.opendatahub.io/facebook-opt-125m-simulated condition met maasmodelref.maas.opendatahub.io/premium-simulated-simulated-premium condition met ✅ Governed MaaSModelRefs ready Waiting for AuthPolicies targeting gateway 'maas-default-gateway' to be enforced (timeout: 180s)... Skipping openshift-ai-inference-authn (targets openshift-ai-inference, not maas-default-gateway) ✅ All AuthPolicies targeting maas-default-gateway enforced (1 policies) 2026-11-02T20:25:57Z deploy_models end Patching Authorino to log_level DEBUG... authorino.operator.authorino.kuadrant.io/authorino patched ✅ Authorino patched to log_level DEBUG deployment "authorino" successfully rolled out ---------------------------------------- Deploying ai-gateway-controller ---------------------------------------- 2026-11-02T20:25:57Z deploy_ai_gateway_controller start Deploying ai-gateway-controller image: quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-146 praxis-extproc image: quay.io/opendatahub/odh-praxis-extproc:odh-stable namespace: opendatahub gateway: openshift-ingress/maas-default-gateway customresourcedefinition.apiextensions.k8s.io/aiguardrails.aigateway.opendatahub.io created serviceaccount/ai-gateway-controller created clusterrole.rbac.authorization.k8s.io/ai-gateway-controller-role created clusterrolebinding.rbac.authorization.k8s.io/ai-gateway-controller-rolebinding created configmap/ai-gateway-controller-parameters created service/ai-gateway-controller-metrics created deployment.apps/ai-gateway-controller created servicemonitor.monitoring.coreos.com/ai-gateway-controller-metrics created networkpolicy.networking.k8s.io/ai-gateway-controller-allow-metrics-traffic created Waiting for deployment "ai-gateway-controller" rollout to finish: 0 of 1 updated replicas are available... deployment "ai-gateway-controller" successfully rolled out Triggering immediate praxis-extproc reconcile ... deployment.apps/ai-gateway-controller restarted Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... deployment "ai-gateway-controller" successfully rolled out Waiting for shared praxis-extproc (quay.io/opendatahub/odh-praxis-extproc:odh-stable) in openshift-ingress: payload-processing and payload-pre-processing (timeout: 300s) ... ✅ shared praxis-extproc ready: openshift-ingress/payload-processing and openshift-ingress/payload-pre-processing Annotating praxis IPP resources opendatahub.io/managed=false so maas-controller skips them ... deployment.apps/payload-processing annotated service/payload-processing annotated destinationrule.networking.istio.io/payload-processing annotated deployment.apps/payload-pre-processing annotated service/payload-pre-processing annotated destinationrule.networking.istio.io/payload-pre-processing annotated envoyfilter.networking.istio.io/payload-processing annotated networkpolicy.networking.k8s.io/payload-processing annotated Verified shared ext_proc dataplane image: openshift-ingress/payload-processing=quay.io/opendatahub/odh-praxis-extproc:odh-stable ai-gateway-controller rollout complete 2026-11-02T20:26:31Z deploy_ai_gateway_controller end ---------------------------------------- Enabling tenant namespace discovery ---------------------------------------- 2026-11-02T20:26:31Z tenant_namespace_discovery start Enabling --enable-tenant-namespace-discovery on maas-controller... maas-controller already has tenant namespace discovery enabled 2026-11-02T20:26:32Z tenant_namespace_discovery end ---------------------------------------- Setting up variables for tests ---------------------------------------- -- Setting up variables for tests -- HOST: maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev MAAS_API_BASE_URL: https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/maas-api ---------------------------------------- Setting up test tokens ---------------------------------------- Setting up premium test token (SA-based, works when oc whoami -t is unavailable)... Creating namespace: premium-users-namespace namespace/premium-users-namespace created Creating service account: premium-service-account serviceaccount/premium-service-account created Patching MaaSAuthPolicy premium-simulator-access to include system:serviceaccount:premium-users-namespace:premium-service-account... maasauthpolicy.maas.opendatahub.io/premium-simulator-access patched Patching MaaSSubscription premium-simulator-subscription to include system:serviceaccount:premium-users-namespace:premium-service-account... maassubscription.maas.opendatahub.io/premium-simulator-subscription patched Waiting for MaaSSubscriptions to reconcile after patch (timeout: 60s)... ✅ Both subscriptions ready: simulator-subscription=Active, premium-simulator-subscription=Active ✅ Premium test token setup complete (E2E_TEST_TOKEN_SA_* exported) Setting up test tokens (admin + regular user)... Current admin session: system:admin (will be preserved) ⚠️ No htpasswd token available - using SA token (admin tests may fail) Creating namespace: maas-admin namespace/maas-admin created Creating service account: tester-admin-user in maas-admin serviceaccount/tester-admin-user created Creating cluster role binding for tester-admin-user clusterrolebinding.rbac.authorization.k8s.io/tester-admin-user-binding created ✅ User setup completed: tester-admin-user (namespace: maas-admin) role.rbac.authorization.k8s.io/maas-admin-e2e created rolebinding.rbac.authorization.k8s.io/maas-admin-e2e-system-serviceaccount-maas-admin-tester-admin-user created auth.services.platform.opendatahub.io/auth patched ✅ Added system:serviceaccounts:maas-admin to Auth CR adminGroups (SA admin fallback) clusterrole.rbac.authorization.k8s.io/maas-admin created rolebinding.rbac.authorization.k8s.io/odh-admins-maas-admin created Creating separate SA token for regular user (required for IDOR tests)... Creating service account: tester-regular-user in default serviceaccount/tester-regular-user created Creating cluster role binding for tester-regular-user clusterrolebinding.rbac.authorization.k8s.io/tester-regular-user-binding created ✅ User setup completed: tester-regular-user (namespace: default) ✅ Regular user token for tester-regular-user (SA-based, namespace: default) Token setup complete (main session unchanged: system:admin) ---------------------------------------- Ensuring gateway allows model HTTPRoutes ---------------------------------------- /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 112: CUSTOM_RESOURCE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 113: NAMESPACE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 114: RESOURCE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 115: CRD_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 116: CSV_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 117: SUBSCRIPTION_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 118: POD_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 119: WEBHOOK_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 120: CUSTOM_CHECK_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 121: AUTHORINO_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 122: ROLLOUT_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 123: KUBECONFIG_WAIT_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 124: CATALOGSOURCE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 125: LLMIS_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 126: MAASMODELREF_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 127: AUTHPOLICY_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 130: _MAX_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 165: LOG_LEVEL_DEBUG: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 166: LOG_LEVEL_INFO: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 167: LOG_LEVEL_WARN: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 168: LOG_LEVEL_ERROR: readonly variable Gateway openshift-ingress/maas-default-gateway already allows HTTPRoutes from all namespaces ---------------------------------------- Validating Deployment ---------------------------------------- 2026-11-02T20:26:37Z validate start Deployment Validation ========================================= 🚀 MaaS Platform Deployment Validation ========================================= ========================================= 1️⃣ Component Status Checks ========================================= 🔍 Checking: MaaS API pods ✅ PASS: MaaS API has 1 running pod(s) 🔍 Checking: Policy engine pods (RHCL/Kuadrant) ✅ PASS: Policy engine has 7 running pod(s) in kuadrant-system 🔍 Checking: OpenDataHub/KServe pods ℹ️ opendatahub namespace: 8 running pod(s) ✅ PASS: OpenDataHub/RHOAI has 8 total running pod(s) 🔍 Checking: LLM namespace and models ✅ PASS: Found 8 LLMInferenceService(s) with 8 running pod(s) ========================================= 2️⃣ Gateway Status ========================================= 🔍 Checking: Gateway resource ✅ PASS: Gateway is Accepted and Programmed 🔍 Checking: HTTPRoute for maas-api ✅ PASS: HTTPRoute maas-api-route is configured and accepted 🔍 Checking: Gateway hostname ✅ PASS: Gateway hostname (from MAAS_GATEWAY_HOST): https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev ========================================= 3️⃣ Policy Status ========================================= 🔍 Checking: AuthPolicy ✅ PASS: AuthPolicy is configured and accepted 🔍 Checking: TokenRateLimitPolicy ✅ PASS: TokenRateLimitPolicy is configured and accepted ========================================= 4️⃣ API Endpoint Tests ========================================= ℹ️ Using gateway endpoint: https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev 🔍 Checking: Authentication token ✅ PASS: OpenShift identity token available 🔍 Checking: MaaS API key creation ✅ PASS: MaaS API key created (name: validate-test-1793651201) 🔍 Checking: Models endpoint ℹ️ Testing: curl -sSk https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/maas-api/v1/models -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" ✅ PASS: Models endpoint accessible, found 1 model(s) ℹ️ Available models: • publishers/llm/models/facebook/opt-125m - https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev ℹ️ Using first available model: publishers/llm/models/facebook/opt-125m for validation 🔍 Checking: Model inference endpoint ℹ️ Testing: curl -sSk -X POST https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/v1/chat/completions -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"model": "publishers/llm/models/facebook/opt-125m", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50}' ✅ PASS: Model inference endpoint working ℹ️ Response: {"id":"chatcmpl-c7592d53-d8a9-52f7-9944-d580d25912ab","created":1793651202,"model":"facebook/opt-125m","usage":{"prompt_tokens":6,"completion_tokens":50,"total_tokens":56},"object":"chat.completion"," 🔍 Checking: Rate limiting ℹ️ User tier: unknown (could not extract from token) ℹ️ Sending 10 rapid requests to test rate limiting... ✅ PASS: Rate limiting is working (2 successful, 8 rate limited) 🔍 Checking: Authorization enforcement (401 without token) ✅ PASS: Authorization is enforced (got 401 without token) ========================================= 📊 Validation Summary ========================================= Results: ✅ Passed: 15 ❌ Failed: 0 ⚠️ Warnings: 0 ✅ PASS: All critical checks passed! 🎉 Next steps: 1. Deploy a model: kustomize build docs/samples/models/simulator | kubectl apply -f - 2. Access the API at: https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev} 3. Check documentation: docs/README.md 4. Re-run validation with specific model: ./scripts/validate-deployment.sh MODEL_NAME Deployment validation completed 2026-11-02T20:26:45Z validate end ---------------------------------------- Running E2E Tests ---------------------------------------- -- E2E Tests (ai-gateway-controller, no external-model tests) -- Waiting for gateway: https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/maas-api/health ... Gateway reachable (HTTP 200) Waiting for authenticated gateway access ... Authenticated gateway access working (HTTP 200) Installing python3.11 (e2e requires Python >= 3.10; system python3 is too old)... Creating Python venv for e2e tests (/usr/bin/python3.11)... WARNING: The directory '/opt/app-root/src/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. WARNING: The directory '/opt/app-root/src/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. WARNING: The directory '/opt/app-root/src/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. Running E2E pass 1/2: parallel (E2E_PARALLEL_WORKERS=7, --dist=loadgroup, -m 'not serial') ============================= test session starts ============================== platform linux -- Python 3.11.13, pytest-9.2.0, pluggy-1.7.0 -- /workspace/source/test/e2e/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.142.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '9.2.0', 'pluggy': '1.7.0'}, 'Plugins': {'xdist': '3.8.0', 'metadata': '3.1.1', 'html': '4.2.0'}, 'PLATFORM': 'el9'} rootdir: /workspace/source/test/maas-e2e/test/e2e configfile: pyproject.toml plugins: xdist-3.8.0, metadata-3.1.1, html-4.2.0 created: 7/7 workers 7 workers [208 items] scheduling tests via LoadGroupScheduling test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_list_api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_other_user_forbidden test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_revoke_api_key test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_admin_manage_other_users_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_non_admin_cannot_access_other_users_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_create_api_key test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_own_keys [gw6] [ 0%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_other_user_forbidden test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_missing_scope_returns_400 [gw6] [ 0%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_missing_scope_returns_400 test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[within-limit] [gw6] [ 1%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[within-limit] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[at-limit] [gw6] [ 1%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[at-limit] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[exceeds-limit] [gw6] [ 2%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[exceeds-limit] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[without-expiration] [gw6] [ 2%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[without-expiration] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[short-expiration] [gw6] [ 3%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyExpiration::test_expiration_policy_boundaries[short-expiration] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_api_key_model_access_success [gw6] [ 3%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_api_key_model_access_success test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_invalid_api_key_rejected [gw6] [ 4%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_invalid_api_key_rejected test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_no_auth_header_rejected [gw6] [ 4%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_no_auth_header_rejected test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_revoked_api_key_rejected [gw6] [ 5%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_revoked_api_key_rejected test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_api_key_chat_completions [gw6] [ 5%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyModelInference::test_api_key_chat_completions test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_double_revoke_returns_404 [gw6] [ 6%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_double_revoke_returns_404 test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_nonexistent_key_returns_404 [gw6] [ 6%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_nonexistent_key_returns_404 test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_then_create_new_key_works [gw6] [ 7%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_then_create_new_key_works test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_individual_revoke_multiple_keys [gw6] [ 7%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_individual_revoke_multiple_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_keys_rejected_at_gateway [gw6] [ 8%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyRevocationE2E::test_revoke_keys_rejected_at_gateway test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_cronjob_exists_and_configured [gw6] [ 8%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_cronjob_exists_and_configured test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_cleanup_networkpolicy_exists [gw6] [ 9%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_cleanup_networkpolicy_exists test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_create_ephemeral_key [gw6] [ 9%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_create_ephemeral_key test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_trigger_cleanup_preserves_active_keys [gw6] [ 10%] SKIPPED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestEphemeralKeyCleanup::test_trigger_cleanup_preserves_active_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_active_subscription [gw6] [ 10%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_active_subscription test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_degraded_subscription [gw4] [ 11%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_non_admin_cannot_access_other_users_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_dry_run_by_subscription [gw3] [ 11%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_admin_manage_other_users_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_dry_run [gw3] [ 12%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_dry_run test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionFilter::test_search_without_subscription_returns_all [gw3] [ 12%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionFilter::test_search_without_subscription_returns_all test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_create_api_key_with_labels [gw3] [ 12%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_create_api_key_with_labels test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_search_api_keys_by_labels [gw3] [ 13%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_search_api_keys_by_labels test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_labels_validation_errors [gw3] [ 13%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_labels_validation_errors test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_backward_compatibility_no_labels [gw3] [ 14%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyLabels::test_backward_compatibility_no_labels test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_subscription_namespace_visible_to_api [gw3] [ 14%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_subscription_namespace_visible_to_api test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_another_namespace_not_visible_to_api [gw3] [ 15%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_another_namespace_not_visible_to_api test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_maas_subscription_namespace [gw3] [ 15%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_maas_subscription_namespace test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_another_namespace [gw3] [ 16%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_another_namespace test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_auth_policy_model_ref [gw3] [ 16%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_auth_policy_model_ref test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_subscription_model_ref [gw3] [ 17%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_subscription_model_ref test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAPIKeyManagementIsolation::test_api_key_cannot_mint_another_api_key [gw3] [ 17%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAPIKeyManagementIsolation::test_api_key_cannot_mint_another_api_key test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[username-only] [gw6] [ 18%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_degraded_subscription test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_failed_subscription [gw3] [ 18%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[username-only] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[group-only] [gw3] [ 19%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[group-only] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[identity-envelope] [gw3] [ 19%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[identity-envelope] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[admin-scope] [gw5] [ 20%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_own_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_combined_user_and_subscription [gw4] [ 20%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_dry_run_by_subscription test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionFilter::test_search_filters_by_subscription [gw3] [ 21%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[admin-scope] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_injected_identity_headers_rejected_on_inference [gw3] [ 21%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_injected_identity_headers_rejected_on_inference test/maas-e2e/test/e2e/tests/test_negative_security.py::TestCrossModelAccess::test_key_cannot_access_model_outside_subscription [gw3] [ 22%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestCrossModelAccess::test_key_cannot_access_model_outside_subscription test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_subscription_with_nonexistent_model_ref [gw0] [ 22%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_create_api_key test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_admin_can_revoke_any_user [gw0] [ 23%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_admin_can_revoke_any_user [gw6] [ 23%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_failed_subscription test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[keyname-only] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderAbuse::test_special_characters_in_subscription_header [gw6] [ 24%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[keyname-only] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[subscriptions-select] [gw0] [ 24%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderAbuse::test_special_characters_in_subscription_header test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-cleanup] [gw6] [ 25%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[subscriptions-select] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-validate] [gw0] [ 25%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-cleanup] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_health_endpoint_accessible [gw0] [ 25%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_health_endpoint_accessible test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_authorized_user_gets_200 [gw6] [ 26%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-validate] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_v1_models_via_maas_api_prefix [gw6] [ 26%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_v1_models_via_maas_api_prefix test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_invalid_token_gets_403 [gw0] [ 27%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_authorized_user_gets_200 test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_no_auth_gets_401 [gw0] [ 27%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_no_auth_gets_401 test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_uses_highest_priority_subscription [gw6] [ 28%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_invalid_token_gets_403 test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_wrong_group_gets_403 [gw6] [ 28%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAuthEnforcement::test_wrong_group_gets_403 test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_nonexistent_subscription_errors [gw0] [ 29%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_uses_highest_priority_subscription test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_with_explicit_simulator_subscription [gw6] [ 29%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_nonexistent_subscription_errors [gw0] [ 30%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestAPIKeySubscriptionBinding::test_create_api_key_with_explicit_simulator_subscription [gw3] [ 30%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_subscription_with_nonexistent_model_ref test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_authpolicy_with_nonexistent_model_ref [gw5] [ 31%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_combined_user_and_subscription test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_duplicate_subscription_headers_ignored [gw3] [ 31%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_authpolicy_with_nonexistent_model_ref test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_both_access_and_subscription_gets_200 [gw2] [ 32%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_revoke_api_key test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_by_subscription_forbidden_for_non_admin [gw2] [ 32%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_by_subscription_forbidden_for_non_admin test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_mint_api_key_denied_for_inaccessible_subscription test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_subscribed_user_gets_200 test/maas-e2e/test/e2e/tests/test_subscription.py::TestOrderingEdgeCases::test_subscription_before_auth_policy [gw5] [ 33%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_duplicate_subscription_headers_ignored test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_subscription_but_no_access_gets_403 [gw6] [ 33%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_subscribed_user_gets_200 test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_subscription_but_no_auth_gets_403 [gw4] [ 34%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionFilter::test_search_filters_by_subscription test/maas-e2e/test/e2e/tests/test_negative_security.py::TestExpiredKeyRejection::test_expired_key_rejected_at_gateway [gw1] [ 34%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_list_api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_by_subscription [gw3] [ 35%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_both_access_and_subscription_gets_200 test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_multiple_subscriptions_separate_keys_gets_200 [gw4] [ 35%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestExpiredKeyRejection::test_expired_key_rejected_at_gateway test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_failed_status_with_missing_model [gw6] [ 36%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_subscription_but_no_auth_gets_403 test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_active_status_with_valid_model [gw5] [ 36%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_subscription_but_no_access_gets_403 test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_failed_status_with_missing_model [gw0] [ 37%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestOrderingEdgeCases::test_subscription_before_auth_policy test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_active_status_with_valid_model [gw2] [ 37%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_mint_api_key_denied_for_inaccessible_subscription test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_access_gets_200 [gw1] [ 37%] PASSED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyBulkOperations::test_bulk_revoke_by_subscription test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_degraded_status_with_partial_models [gw3] [ 38%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_multiple_subscriptions_separate_keys_gets_200 test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_degraded_status_with_partial_models [gw2] [ 38%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_access_gets_200 test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_returns_accessible_subscriptions [gw4] [ 39%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_failed_status_with_missing_model test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_status_transitions_on_model_deletion [gw4] [ 39%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_status_transitions_on_model_deletion test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_model_ref_display_name_and_description_enriched [gw4] [ 40%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_model_ref_display_name_and_description_enriched test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_returns_subscriptions_for_model [gw6] [ 40%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_active_status_with_valid_model test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_degraded_healthy_model_allows_inference [gw4] [ 41%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_returns_subscriptions_for_model test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_unknown_model_returns_empty [gw4] [ 41%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptionsForModel::test_unknown_model_returns_empty test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestSubscriptionModelAccessFiltering::test_omits_subscription_with_no_authorized_models [gw6] [ 42%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_degraded_healthy_model_allows_inference test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestSubscriptionModelAccessFiltering::test_filters_unauthorized_models [gw5] [ 42%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_failed_status_with_missing_model test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_models_endpoint_with_degraded_subscription_api_key [gw4] [ 43%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestSubscriptionModelAccessFiltering::test_omits_subscription_with_no_authorized_models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_empty_subscription_header_value [gw4] [ 43%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_empty_subscription_header_value test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_user_token_returns_all_models [gw1] [ 44%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_degraded_status_with_partial_models test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_unauthenticated_returns_401[all-subscriptions] [gw6] [ 44%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestSubscriptionModelAccessFiltering::test_filters_unauthorized_models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_deduplication_same_model_multiple_refs [gw0] [ 45%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_active_status_with_valid_model test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_models_endpoint_with_degraded_subscription_kube_token [gw5] [ 45%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_models_endpoint_with_degraded_subscription_api_key test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_distinct_models_in_subscription [gw1] [ 46%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_unauthenticated_returns_401[all-subscriptions] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_response_schema_and_metadata [gw2] [ 46%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_returns_accessible_subscriptions test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_subscription_includes_model_refs [gw1] [ 47%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_response_schema_and_metadata test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_invalid_subscription_header_403 [gw0] [ 47%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestDegradedSubscriptionFiltering::test_models_endpoint_with_degraded_subscription_kube_token test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_with_deleted_subscription_403 [gw4] [ 48%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_user_token_returns_all_models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_user_token_with_subscription_header_filters [gw2] [ 48%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_subscription_includes_model_refs test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_access_denied_to_subscription_403 [gw6] [ 49%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_deduplication_same_model_multiple_refs test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_scoped_to_subscription [gw3] [ 49%] PASSED test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_authpolicy_degraded_status_with_partial_models test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_unauthenticated_returns_401[subscriptions-for-model] [gw1] [ 50%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_invalid_subscription_header_403 test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_ignores_subscription_header [gw3] [ 50%] PASSED test/maas-e2e/test/e2e/tests/test_subscription_list_endpoints.py::TestListSubscriptions::test_unauthenticated_returns_401[subscriptions-for-model] test/maas-e2e/test/e2e/tests/test_smoke.py::test_healthz_or_404 [gw3] [ 50%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_healthz_or_404 test/maas-e2e/test/e2e/tests/test_smoke.py::test_models_catalog [gw3] [ 51%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_models_catalog test/maas-e2e/test/e2e/tests/test_smoke.py::test_chat_completions_gateway_alive [gw3] [ 51%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_chat_completions_gateway_alive test/maas-e2e/test/e2e/tests/test_smoke.py::test_chat_completions_streaming_sse [gw3] [ 52%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_chat_completions_streaming_sse test/maas-e2e/test/e2e/tests/test_smoke.py::test_legacy_completions_optionally [gw3] [ 52%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_legacy_completions_optionally test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_tenant_ready_and_phase_healthy [gw3] [ 53%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_tenant_ready_and_phase_healthy test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_status_has_phase_and_conditions [gw5] [ 53%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_distinct_models_in_subscription test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_with_inaccessible_subscription_403 [gw3] [ 54%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_status_has_phase_and_conditions test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_spec_is_well_formed [gw3] [ 54%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_spec_is_well_formed test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantNoFalseOwnership::test_maas_user_crs_not_owned_by_tenant [gw3] [ 55%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantNoFalseOwnership::test_maas_user_crs_not_owned_by_tenant test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_default_exists [gw4] [ 55%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_user_token_with_subscription_header_filters test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_service_account_token_multiple_subs_no_header [gw3] [ 56%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_default_exists test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_not_terminating [gw3] [ 56%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_not_terminating test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_tenant_config_lists_config_owner_reference [gw3] [ 57%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_tenant_config_lists_config_owner_reference test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_maas_controller_deployment_does_not_list_config_owner_reference [gw3] [ 57%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_maas_controller_deployment_does_not_list_config_owner_reference test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_requires_auth [gw6] [ 58%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_scoped_to_subscription test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_unauthenticated_request_401 [gw6] [ 58%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_unauthenticated_request_401 test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_authenticated [gw6] [ 59%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_authenticated test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_gateway_matches_deployment [gw6] [ 59%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_gateway_matches_deployment test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_not_exposed_through_gateway [gw6] [ 60%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_not_exposed_through_gateway test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_default_aitenant_bootstraps_maas_tenant_config_without_gateway_mutation [gw3] [ 60%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_requires_auth test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_with_invalid_token [gw6] [ 61%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_default_aitenant_bootstraps_maas_tenant_config_without_gateway_mutation test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_rejected_outside_ai_tenants_namespace [gw3] [ 61%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_with_invalid_token test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_create_bootstrap_resources [gw6] [ 62%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_rejected_outside_ai_tenants_namespace test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_migrates_and_removes_legacy_tenant [gw5] [ 62%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_with_inaccessible_subscription_403 test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_conditions_use_kubernetes_metav1_shape [gw2] [ 62%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_access_denied_to_subscription_403 test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_service_account_token_multiple_subs_with_header [gw1] [ 63%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_ignores_subscription_header test/maas-e2e/test/e2e/tests/test_smoke.py::test_tokens_endpoint_replaced_by_api_keys [gw1] [ 63%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_tokens_endpoint_replaced_by_api_keys test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_label_removal_stops_reconciliation [gw5] [ 64%] ERROR test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_conditions_use_kubernetes_metav1_shape test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_derives_non_default_tenant_namespace [gw4] [ 64%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_service_account_token_multiple_subs_no_header test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_default_aitenant_lists_config_owner_reference [gw4] [ 65%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_default_aitenant_lists_config_owner_reference test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_per_tenant_oidc_configuration [gw4] [ 65%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_per_tenant_oidc_configuration test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_namespace_qualified_collision_prevention [gw0] [ 66%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_api_key_with_deleted_subscription_403 test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_api_keys_different_subscriptions [gw2] [ 66%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_service_account_token_multiple_subs_with_header test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_labeled_tenant_namespace_is_discovered [gw0] [ 67%] PASSED test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_api_keys_different_subscriptions test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantWebhookValidation::test_maassubscription_rejected_without_tenant_config_cr [gw0] [ 67%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantWebhookValidation::test_maassubscription_rejected_without_tenant_config_cr test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestLegacyDefaultNamespaceStillWorks::test_models_as_a_service_namespace_reconciles [gw1] [ 68%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_label_removal_stops_reconciliation test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_unlabeled_namespace_ignored [gw3] [ 68%] FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_create_bootstrap_resources test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects [gw6] [ 69%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_migrates_and_removes_legacy_tenant test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_delete_releases_tenantconfig_finalizer [gw5] [ 69%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_derives_non_default_tenant_namespace test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_dynamic_discovery_after_label_added [gw4] [ 70%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_namespace_qualified_collision_prevention test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_tenant_admin_rbac_is_namespace_scoped [gw1] [ 70%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_unlabeled_namespace_ignored test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_cross_tenant_isolation [gw0] [ 71%] FAILED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestLegacyDefaultNamespaceStillWorks::test_models_as_a_service_namespace_reconciles test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_same_tenant_access [gw2] [ 71%] FAILED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_labeled_tenant_namespace_is_discovered test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantWebhookValidation::test_maasauthpolicy_rejected_without_tenant_config_cr [gw3] [ 72%] FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_unauthorized_access [gw6] [ 72%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_delete_releases_tenantconfig_finalizer test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_each_tenant_returns_own_gateway [gw2] [ 73%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantWebhookValidation::test_maasauthpolicy_rejected_without_tenant_config_cr test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_gateway_default_auth_scoped_if_present [gw2] [ 73%] SKIPPED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_gateway_default_auth_scoped_if_present [gw5] [ 74%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_dynamic_discovery_after_label_added test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyStructure::test_gateway_auth_policy_contract [gw5] [ 74%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyStructure::test_gateway_auth_policy_contract [gw4] [ 75%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_tenant_admin_rbac_is_namespace_scoped test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyLifecycle::test_gateway_auth_rego_is_fixed_size [gw4] [ 75%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyLifecycle::test_gateway_auth_rego_is_fixed_size [gw1] [ 75%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_cross_tenant_isolation test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyLifecycle::test_only_one_gateway_authpolicy_named_maas_gateway_auth [gw1] [ 76%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyLifecycle::test_only_one_gateway_authpolicy_named_maas_gateway_auth [gw0] [ 76%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_same_tenant_access test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_management_rules_have_model_identity_guards [gw0] [ 77%] PASSED test/maas-e2e/test/e2e/tests/test_gateway_scoped_authpolicy.py::TestGatewayAuthPolicyManagementEndpointAccess::test_management_rules_have_model_identity_guards [gw6] [ 77%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_each_tenant_returns_own_gateway [gw3] [ 78%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery_isolation.py::test_tenant_discovery_unauthorized_access ==================================== ERRORS ==================================== _ ERROR at setup of TestTenantContract.test_conditions_use_kubernetes_metav1_shape _ [gw5] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module", autouse=True) def require_tenant_crd(): r = _oc_run(["get", "crd", TENANT_CRD]) if r.returncode != 0: if _oc_output_not_found(r): pytest.skip( f"Missing CRD {TENANT_CRD} (transitional skip: install maas-controller manifests " f"so CRDs exist; then controller creates {TENANT_NAME})." ) combined = (r.stderr or "") + (r.stdout or "") > pytest.fail(f"`oc get crd {TENANT_CRD}` failed: {combined.strip()}") E Failed: `oc get crd maastenantconfigs.maas.opendatahub.io` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/test_tenant.py:95: Failed =================================== FAILURES =================================== ________ TestAITenantLifecycle.test_aitenant_create_bootstrap_resources ________ [gw3] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python self = def test_aitenant_create_bootstrap_resources(self): case = _new_aitenant_case() try: _apply_gateway_fixture(case) _apply_aitenant(case) > _assert_aitenant_bootstrap_resources(case) test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:479: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:262: in _assert_aitenant_bootstrap_resources aitenant = _wait_for_json( test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:164: in _wait_for_json obj = _get_json_or_none(kind, name, namespace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-ait-2b3e20d8', namespace = 'ai-tenants' def _get_json_or_none(kind, name, namespace=None): args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-ait-2b3e20d8 -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:157: RuntimeError _ TestLegacyDefaultNamespaceStillWorks.test_models_as_a_service_namespace_reconciles _ [gw0] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python self = def test_models_as_a_service_namespace_reconciles(self): ns = legacy_default_namespace() suffix = uuid.uuid4().hex[:8] policy_name = f"e2e-default-ns-{suffix}" try: apply_maas_auth_policy(policy_name, ns) wait_for_finalizer("maasauthpolicy", policy_name, ns, FINALIZER_AUTHPOLICY) > wait_for_status_phase("maasauthpolicy", policy_name, ns, expected_phase="Active") test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py:394: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'e2e-default-ns-b46dc8fb' namespace = 'models-as-a-service' predicate = ._predicate at 0x7fd28be01da0> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/e2e-default-ns-b46dc8fb in models-as-a-service did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"e2e-default-ns-b46dc8fb","namespace":"models-as-a-service"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-11-02T20:38:49Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'e2e-default-ns-b46dc8fb', 'namespace': 'models-as-a-service', 'resourceVersion': '51468', 'uid': '30b8bcf2-2f86-4f9a-b02f-4824615da462'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-11-02T20:38:49Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/maas-gateway-auth to be accepted and enforced: AuthPolicy status observed generation 3, current generation is 36', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError ___ TestTenantNamespaceDiscovery.test_labeled_tenant_namespace_is_discovered ___ [gw2] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python self = def test_labeled_tenant_namespace_is_discovered(self): """1.1: Controller discovers labeled tenant namespaces and reconciles CRs.""" case = new_discovery_case(use_default_gateway=True) try: apply_discovery_labels(case["tenant_ns"], case["tenant_label_name"]) apply_tenant_cr(case["tenant_ns"], DEFAULT_GATEWAY_NAME) apply_maas_auth_policy(case["policy_name"], case["tenant_ns"]) apply_maas_subscription(case["subscription_name"], case["tenant_ns"]) wait_for_finalizer("maasauthpolicy", case["policy_name"], case["tenant_ns"], FINALIZER_AUTHPOLICY) wait_for_finalizer("maassubscription", case["subscription_name"], case["tenant_ns"], FINALIZER_SUBSCRIPTION) > auth = wait_for_status_phase( "maasauthpolicy", case["policy_name"], case["tenant_ns"], expected_phase="Active", ) test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py:92: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:343: in wait_for_status_phase return wait_for_json(kind, name, namespace, predicate=_predicate, timeout=timeout, interval=interval) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'maasauthpolicy', name = 'e2e-policy-21eab0e9' namespace = 'ai-tenant-e2e-mt-21eab0e9' predicate = ._predicate at 0x7fcc912d5620> timeout = 180, interval = 5 def wait_for_json( kind: str, name: str, namespace: Optional[str] = None, *, predicate=None, timeout: int = 180, interval: int = 5, ) -> dict: deadline = time.time() + timeout last_obj = None while time.time() < deadline: obj = get_json_or_none(kind, name, namespace) if obj is not None: last_obj = obj if predicate is None or predicate(obj): return obj time.sleep(interval) > raise AssertionError( f"{kind}/{name} in {namespace or ''} did not satisfy condition. Last object: {last_obj}" ) E AssertionError: maasauthpolicy/e2e-policy-21eab0e9 in ai-tenant-e2e-mt-21eab0e9 did not satisfy condition. Last object: {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"maas.opendatahub.io/v1alpha1","kind":"MaaSAuthPolicy","metadata":{"annotations":{},"name":"e2e-policy-21eab0e9","namespace":"ai-tenant-e2e-mt-21eab0e9"},"spec":{"modelRefs":[{"name":"facebook-opt-125m-simulated","namespace":"llm"}],"subjects":{"groups":[{"name":"system:authenticated"}]}}}\n'}, 'creationTimestamp': '2026-11-02T20:38:09Z', 'finalizers': ['maas.opendatahub.io/authpolicy-cleanup'], 'generation': 1, 'name': 'e2e-policy-21eab0e9', 'namespace': 'ai-tenant-e2e-mt-21eab0e9', 'resourceVersion': '50471', 'uid': '2e1622b9-15a7-4062-b96e-9cb83efb4100'}, 'spec': {'modelRefs': [{'name': 'facebook-opt-125m-simulated', 'namespace': 'llm'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}, 'status': {'conditions': [{'lastTransitionTime': '2026-11-02T20:38:09Z', 'message': 'Waiting for gateway AuthPolicy openshift-ingress/maas-gateway-auth to be accepted and enforced: AuthPolicy status observed generation 3, current generation is 31', 'observedGeneration': 1, 'reason': 'Unknown', 'status': 'Unknown', 'type': 'Ready'}], 'phase': 'Pending'}} test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:288: AssertionError ----------------------------- Captured stdout call ----------------------------- [cleanup] failed to delete namespace/ai-tenant-e2e-mt-21eab0e9: `oc delete namespace ai-tenant-e2e-mt-21eab0e9 --ignore-not-found --timeout=90s` failed: error: timed out waiting for the condition on namespaces/ai-tenant-e2e-mt-21eab0e9 _ TestAITenantLifecycle.test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects _ [gw3] linux -- Python 3.11.13 /workspace/source/test/e2e/.venv/bin/python self = def test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects(self): case = _new_aitenant_case() try: _apply_gateway_fixture(case) _apply_aitenant(case) _assert_aitenant_bootstrap_resources(case) _apply( { "apiVersion": "v1", "kind": "Secret", "metadata": {"name": case["user_secret"], "namespace": case["tenant_ns"]}, "type": "Opaque", "stringData": {"purpose": "unrelated-user-content"}, } ) _apply( { "apiVersion": "rbac.authorization.k8s.io/v1", "kind": "RoleBinding", "metadata": {"name": case["user_rolebinding"], "namespace": case["tenant_ns"]}, "roleRef": { "apiGroup": "rbac.authorization.k8s.io", "kind": "Role", "name": case["tenant_admin_role"], }, "subjects": [ { "apiGroup": "rbac.authorization.k8s.io", "kind": "User", "name": "e2e-preserved-user", } ], } ) _delete_aitenant(case) _wait_for_not_found(TENANT_CONFIG_KIND, TENANT_NAME, case["tenant_ns"]) _wait_for_not_found("role", case["tenant_admin_role"], case["tenant_ns"]) > _wait_for_not_found("role", case["object_admin_role"], AITENANT_NAMESPACE) test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:633: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:176: in _wait_for_not_found if _get_json_or_none(kind, name, namespace) is None: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'role', name = 'aitenant-e2e-ait-df17143f-object-admin' namespace = 'ai-tenants' def _get_json_or_none(kind, name, namespace=None): args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get role aitenant-e2e-ait-df17143f-object-admin -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:157: RuntimeError ------ generated xml file: /workspace/artifacts-dir/e2e-system:admin.xml ------- - Generated html report: file:///workspace/artifacts-dir/e2e-system%3Aadmin.html - =========================== short test summary info ============================ FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_create_bootstrap_resources FAILED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestLegacyDefaultNamespaceStillWorks::test_models_as_a_service_namespace_reconciles FAILED test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_labeled_tenant_namespace_is_discovered FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects ERROR test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_conditions_use_kubernetes_metav1_shape !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 5 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! = 4 failed, 144 passed, 14 skipped, 123 warnings, 1 error in 1379.21s (0:22:59) = Running E2E pass 2/2: serial cluster mutators (-m serial, single worker) ============================= test session starts ============================== platform linux -- Python 3.11.13, pytest-9.2.0, pluggy-1.7.0 -- /workspace/source/test/e2e/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.142.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '9.2.0', 'pluggy': '1.7.0'}, 'Plugins': {'xdist': '3.8.0', 'metadata': '3.1.1', 'html': '4.2.0'}, 'PLATFORM': 'el9'} rootdir: /workspace/source/test/maas-e2e/test/e2e configfile: pyproject.toml plugins: xdist-3.8.0, metadata-3.1.1, html-4.2.0 collecting ... collected 251 items / 208 deselected / 43 selected test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_pending_subscription [token] using env TOKEN (masked): 1037 PASSED [ 2%] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_reject_key_for_unreconciled_subscription FAILED [ 4%] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAuthPolicyRemoval::test_authpolicy_deletion_revokes_access FAILED [ 6%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_auth_pass_no_subscription_gets_403 FAILED [ 9%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_rate_limit_exhaustion_gets_429 FAILED [ 11%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_models_endpoint_exempt_from_rate_limiting FAILED [ 13%] =================================== FAILURES =================================== __ TestAPIKeySubscriptionPhases.test_reject_key_for_unreconciled_subscription __ self = @pytest.mark.serial def test_reject_key_for_unreconciled_subscription(self): """ API key creation is rejected for unreconciled subscription (empty phase). Note: Temporarily sets webhook failurePolicy to Ignore to allow creating resources while controller is down, then restores to Fail. """ ns = _ns() subscription_name = "e2e-apikey-unreconciled-sub" auth_name = "e2e-apikey-unreconciled-auth" sa_name = "e2e-apikey-unreconciled-sa" webhook_name = "maas-validating-webhook-configuration" try: # Create service account and get token oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) # Temporarily set webhook failurePolicy to Ignore # This allows creates to succeed when controller/webhook is unavailable # Find webhook indices dynamically by name to avoid brittleness result = subprocess.run( ["oc", "get", "validatingwebhookconfiguration", webhook_name, "-o", "json"], capture_output=True, text=True, check=True ) webhook_config = json.loads(result.stdout) patch_ops = [] for idx, webhook in enumerate(webhook_config.get("webhooks", [])): if webhook.get("name") in ["vmaassubscription.kb.io", "vmaasauthpolicy.kb.io"]: patch_ops.append({"op": "replace", "path": f"/webhooks/{idx}/failurePolicy", "value": "Ignore"}) subprocess.run( ["oc", "patch", "validatingwebhookconfiguration", webhook_name, "--type=json", "-p", json.dumps(patch_ops)], capture_output=True, text=True, check=True ) # Scale down controller to prevent reconciliation _scale_controller_down() # Create resources (webhook unavailable but Ignore policy allows creates) _create_test_auth_policy(auth_name, MODEL_REF, users=[sa_user]) _create_test_subscription(subscription_name, MODEL_REF, users=[sa_user]) # Verify subscription is unreconciled (empty phase) cr = _get_cr("maassubscription", subscription_name, namespace=ns) phase = cr.get("status", {}).get("phase", "") assert phase == "", f"Expected empty phase, got: {phase}" log.info("✅ Subscription is unreconciled (empty phase)") # Try to create API key (should fail with 400) response = requests.post( f"{_maas_api_url()}/v1/api-keys", headers={ "Authorization": f"Bearer {oc_token}", "Content-Type": "application/json" }, json={ "name": "unreconciled-sub-test", "subscription": subscription_name }, timeout=TIMEOUT, verify=TLS_VERIFY, ) assert response.status_code == 400, \ f"Expected 400 for unreconciled subscription, got {response.status_code}: {response.text}" response_data = response.json() assert "code" in response_data and response_data["code"] == "subscription_not_ready", \ f"Expected subscription_not_ready error code, got: {response_data}" log.info("✅ API key creation rejected for unreconciled subscription") finally: # Restore webhook failurePolicy to Fail # Wrap in try/except to guarantee controller scale-up and cleanup happen webhook_restore_error = None try: result = subprocess.run( ["oc", "get", "validatingwebhookconfiguration", webhook_name, "-o", "json"], capture_output=True, text=True ) if result.returncode != 0: raise RuntimeError( f"Failed to get webhook configuration {webhook_name} for restoration: {result.stderr}" ) webhook_config = json.loads(result.stdout) patch_ops = [] for idx, webhook in enumerate(webhook_config.get("webhooks", [])): if webhook.get("name") in ["vmaassubscription.kb.io", "vmaasauthpolicy.kb.io"]: patch_ops.append({"op": "replace", "path": f"/webhooks/{idx}/failurePolicy", "value": "Fail"}) if not patch_ops: raise RuntimeError( f"No matching webhooks found to restore in {webhook_name}. " f"Expected vmaassubscription.kb.io and vmaasauthpolicy.kb.io. " f"Found: {[w.get('name') for w in webhook_config.get('webhooks', [])]}" ) restore_result = subprocess.run( ["oc", "patch", "validatingwebhookconfiguration", webhook_name, "--type=json", "-p", json.dumps(patch_ops)], capture_output=True, text=True ) if restore_result.returncode != 0: raise RuntimeError( f"Webhook left in Ignore state! Failed to restore webhook {webhook_name} " f"with patch_ops {patch_ops}: {restore_result.stderr}" ) except Exception as e: webhook_restore_error = e log.error(f"Exception during webhook restoration: {e}") # Always scale controller back up and cleanup, even if webhook restore failed > _scale_controller_up() test/maas-e2e/test/e2e/tests/test_api_keys.py:1798: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_helper.py:2121: in _scale_controller_up _scale_controller(1, namespace, timeout) test/maas-e2e/test/e2e/tests/test_helper.py:2073: in _scale_controller subprocess.run( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ input = None, capture_output = True, timeout = 60, check = True popenargs = (['oc', 'scale', 'deployment', 'maas-controller', '--replicas=1', '-n', ...],) kwargs = {'text': True, 'stdout': -1, 'stderr': -1} def run(*popenargs, input=None, capture_output=False, timeout=None, check=False, **kwargs): """Run command with arguments and return a CompletedProcess instance. The returned instance will have attributes args, returncode, stdout and stderr. By default, stdout and stderr are not captured, and those attributes will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them, or pass capture_output=True to capture both. If check is True and the exit code was non-zero, it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute, and output & stderr attributes if those streams were captured. If timeout is given, and the process takes too long, a TimeoutExpired exception will be raised. There is an optional argument "input", allowing you to pass bytes or a string to the subprocess's stdin. If you use this argument you may not also use the Popen constructor's "stdin" argument, as it will be used internally. By default, all communication is in bytes, and therefore any "input" should be bytes, and the stdout and stderr will be bytes. If in text mode, any "input" should be a string, and stdout and stderr will be strings decoded according to locale encoding, or by "encoding" if set. Text mode is triggered by setting any of text, encoding, errors or universal_newlines. The other arguments are the same as for the Popen constructor. """ if input is not None: if kwargs.get('stdin') is not None: raise ValueError('stdin and input arguments may not both be used.') kwargs['stdin'] = PIPE if capture_output: if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None: raise ValueError('stdout and stderr arguments may not be used ' 'with capture_output.') kwargs['stdout'] = PIPE kwargs['stderr'] = PIPE with Popen(*popenargs, **kwargs) as process: try: stdout, stderr = process.communicate(input, timeout=timeout) except TimeoutExpired as exc: process.kill() if _mswindows: # Windows accumulates the output in a single blocking # read() call run on child threads, with the timeout # being done in a join() on those threads. communicate() # _after_ kill() is required to collect that and add it # to the exception. exc.stdout, exc.stderr = process.communicate() else: # POSIX _communicate already populated the output so # far into the TimeoutExpired exception. process.wait() raise except: # Including KeyboardInterrupt, communicate handled that. process.kill() # We don't call process.wait() as .__exit__ does that for us. raise retcode = process.poll() if check and retcode: > raise CalledProcessError(retcode, process.args, output=stdout, stderr=stderr) E subprocess.CalledProcessError: Command '['oc', 'scale', 'deployment', 'maas-controller', '--replicas=1', '-n', 'opendatahub']' returned non-zero exit status 1. /usr/lib64/python3.11/subprocess.py:571: CalledProcessError ------------------------------ Captured log call ------------------------------- INFO test_helper:test_helper.py:2070 Scaling maas-controller to 0 replicas in namespace opendatahub... INFO test_helper:test_helper.py:2094 ✓ maas-controller scaled down to 0 replicas INFO test_helper:test_helper.py:1101 Creating MaaSAuthPolicy: e2e-apikey-unreconciled-auth INFO test_helper:test_helper.py:1170 Creating MaaSSubscription: e2e-apikey-unreconciled-sub INFO test_api_keys:test_api_keys.py:1732 ✅ Subscription is unreconciled (empty phase) INFO test_api_keys:test_api_keys.py:1754 ✅ API key creation rejected for unreconciled subscription INFO test_helper:test_helper.py:2070 Scaling maas-controller to 1 replicas in namespace opendatahub... ________ TestAuthPolicyRemoval.test_authpolicy_deletion_revokes_access _________ self = @pytest.mark.serial def test_authpolicy_deletion_revokes_access(self): """Create auth policy, delete it, verify legacy per-model AuthPolicy is absent. Uses the unconfigured model to avoid interfering with other tests. In gateway-only mode, the controller should not create per-model Kuadrant AuthPolicies. This verifies the legacy per-model AuthPolicy remains absent before and after MaaSAuthPolicy deletion. This tests the controller's cleanup logic. Gateway enforcement of AuthPolicy is already covered by other tests (e.g. test_wrong_group_gets_403). """ suffix = uuid.uuid4().hex[:8] policy_name = f"e2e-neg-policy-{suffix}" model_ref = UNCONFIGURED_MODEL_REF kuadrant_auth_name = f"maas-auth-{model_ref}" try: # Create auth policy granting access > _create_test_auth_policy( policy_name, model_ref, groups=["system:authenticated"], ) test/maas-e2e/test/e2e/tests/test_negative_security.py:467: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_helper.py:1102: in _create_test_auth_policy _apply_cr({ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ cr_dict = {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'name': 'e2e-neg-policy-056a59bf...ok-opt-125m-simulated', 'namespace': 'llm'}], 'subjects': {'users': [], 'groups': [{'name': 'system:authenticated'}]}}} def _apply_cr(cr_dict): result = _run_oc(["oc", "apply", "-f", "-"], input_text=json.dumps(cr_dict)) if result.returncode != 0: meta = cr_dict.get("metadata", {}) > raise RuntimeError( f"`oc apply` failed for {cr_dict.get('kind')} " f"{meta.get('namespace', '')}/{meta.get('name')}: " f"{result.stderr.strip() or result.stdout.strip()}" ) E RuntimeError: `oc apply` failed for MaaSAuthPolicy models-as-a-service/e2e-neg-policy-056a59bf: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" test/maas-e2e/test/e2e/tests/test_helper.py:767: RuntimeError ------------------------------ Captured log call ------------------------------- INFO test_helper:test_helper.py:1101 Creating MaaSAuthPolicy: e2e-neg-policy-056a59bf WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 1), retrying in 2s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 2), retrying in 4s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 3), retrying in 6s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 4), retrying in 8s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 5), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 6), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 7), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 8), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 9), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 10), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 11), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 12), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 13), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 14), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 15), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 16), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 17), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 18), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 19), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" _____ TestSubscriptionEnforcement.test_auth_pass_no_subscription_gets_403 ______ self = @pytest.mark.serial def test_auth_pass_no_subscription_gets_403(self): """API key with auth pass but no matching subscription should get 403. The AuthPolicy includes a subscription-error-check rule that calls /internal/v1/subscriptions/select. If no subscription matches the user's groups, the request is denied with 403 "no matching subscription found for user". To test this, we temporarily add system:authenticated to the premium model's AuthPolicy (so auth passes) but keep the subscription only for premium-user (so subscription check fails). """ ns = _ns() api_key = _get_default_api_key() # First verify that default key currently gets 403 on premium model (auth fails) r = _inference(api_key, path=PREMIUM_MODEL_PATH) assert r.status_code == 403, f"Expected 403 for premium model (auth should fail), got {r.status_code}" # Now temporarily add system:authenticated to premium model's AuthPolicy try: # Get current auth policy and add system:authenticated group > _apply_cr({ "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSAuthPolicy", "metadata": {"name": "e2e-auth-pass-sub-fail", "namespace": ns}, "spec": { "modelRefs": [{"name": PREMIUM_MODEL_REF, "namespace": MODEL_NAMESPACE}], "subjects": { "groups": [{"name": "system:authenticated"}], # Auth will pass }, }, }) test/maas-e2e/test/e2e/tests/test_subscription.py:582: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ cr_dict = {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'name': 'e2e-auth-pass-sub-fail'...remium-simulated-simulated-premium', 'namespace': 'llm'}], 'subjects': {'groups': [{'name': 'system:authenticated'}]}}} def _apply_cr(cr_dict): result = _run_oc(["oc", "apply", "-f", "-"], input_text=json.dumps(cr_dict)) if result.returncode != 0: meta = cr_dict.get("metadata", {}) > raise RuntimeError( f"`oc apply` failed for {cr_dict.get('kind')} " f"{meta.get('namespace', '')}/{meta.get('name')}: " f"{result.stderr.strip() or result.stdout.strip()}" ) E RuntimeError: `oc apply` failed for MaaSAuthPolicy models-as-a-service/e2e-auth-pass-sub-fail: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" test/maas-e2e/test/e2e/tests/test_helper.py:767: RuntimeError ------------------------------ Captured log call ------------------------------- INFO test_helper:test_helper.py:514 Using TOKEN env var for API key operations INFO test_helper:test_helper.py:739 Created API key 'e2e-default-key-main' bound to subscription 'simulator-subscription' WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 1), retrying in 2s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 2), retrying in 4s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 3), retrying in 6s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 4), retrying in 8s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 5), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 6), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 7), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 8), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 9), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 10), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 11), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 12), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 13), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 14), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 15), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 16), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 17), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 18), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 19), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" _______ TestSubscriptionEnforcement.test_rate_limit_exhaustion_gets_429 ________ self = @pytest.mark.serial def test_rate_limit_exhaustion_gets_429(self): """ Test that a user gets 429 when they actually exceed their token rate limit. This test creates a dedicated subscription with a very low token limit, sends enough requests to exhaust it, and verifies a 429 response. Uses the unconfigured model to avoid interfering with other tests. """ # Use unconfigured model to isolate this test model_ref = UNCONFIGURED_MODEL_REF model_path = UNCONFIGURED_MODEL_PATH # Create unique subscription and auth policy names auth_policy_name = "e2e-rate-limit-test-auth" subscription_name = "e2e-rate-limit-test-subscription" # Low limit so we exhaust it quickly. Actual tokens consumed per # response are non-deterministic (max_tokens is a ceiling, not exact), # so we send enough requests to be confident we hit the limit without # asserting exactly when the 429 arrives. token_limit = 10 window = "1m" total_requests = 15 try: # 1. Create auth policy allowing system:authenticated > _create_test_auth_policy( name=auth_policy_name, model_refs=[model_ref], groups=["system:authenticated"] ) test/maas-e2e/test/e2e/tests/test_subscription.py:635: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_helper.py:1102: in _create_test_auth_policy _apply_cr({ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ cr_dict = {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'name': 'e2e-rate-limit-test-aut...ok-opt-125m-simulated', 'namespace': 'llm'}], 'subjects': {'users': [], 'groups': [{'name': 'system:authenticated'}]}}} def _apply_cr(cr_dict): result = _run_oc(["oc", "apply", "-f", "-"], input_text=json.dumps(cr_dict)) if result.returncode != 0: meta = cr_dict.get("metadata", {}) > raise RuntimeError( f"`oc apply` failed for {cr_dict.get('kind')} " f"{meta.get('namespace', '')}/{meta.get('name')}: " f"{result.stderr.strip() or result.stdout.strip()}" ) E RuntimeError: `oc apply` failed for MaaSAuthPolicy models-as-a-service/e2e-rate-limit-test-auth: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" test/maas-e2e/test/e2e/tests/test_helper.py:767: RuntimeError ------------------------------ Captured log call ------------------------------- INFO test_helper:test_helper.py:1101 Creating MaaSAuthPolicy: e2e-rate-limit-test-auth WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 1), retrying in 2s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 2), retrying in 4s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 3), retrying in 6s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 4), retrying in 8s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 5), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 6), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 7), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 8), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 9), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 10), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 11), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 12), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 13), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 14), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 15), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 16), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 17), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 18), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 19), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" INFO test_subscription:test_subscription.py:720 Cleaned up rate limit test resources __ TestSubscriptionEnforcement.test_models_endpoint_exempt_from_rate_limiting __ self = @pytest.mark.serial def test_models_endpoint_exempt_from_rate_limiting(self): """ Test that /v1/models endpoint remains accessible when token quota is exhausted. This verifies that users can discover model capabilities even when they've used all their inference tokens. The /v1/models endpoint is a discovery/metadata endpoint that does not consume tokens and should remain accessible. Ref: https://issues.redhat.com/browse/RHOAIENG-46770 Test steps: 1. Create subscription with very low token limit (15 tokens) 2. Exhaust the limit with inference requests (5 requests × 3 tokens = 15) 3. Verify inference requests get 429 (rate limited) 4. Verify /v1/models endpoint still returns 200 (not rate limited) """ # Use unconfigured model to isolate this test model_ref = UNCONFIGURED_MODEL_REF model_path = UNCONFIGURED_MODEL_PATH # Create unique subscription and auth policy names auth_policy_name = "e2e-models-exempt-test-auth" subscription_name = "e2e-models-exempt-test-subscription" # Very low limit for fast, deterministic test # With 3 token limit and max_tokens=1, we're guaranteed to exhaust quota within 5 requests # (even if each request uses exactly 1 token: 5 requests > 3 token limit) token_limit = 3 window = "1m" try: # 1. Create auth policy allowing system:authenticated > _create_test_auth_policy( name=auth_policy_name, model_refs=[model_ref], groups=["system:authenticated"] ) test/maas-e2e/test/e2e/tests/test_subscription.py:755: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_helper.py:1102: in _create_test_auth_policy _apply_cr({ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ cr_dict = {'apiVersion': 'maas.opendatahub.io/v1alpha1', 'kind': 'MaaSAuthPolicy', 'metadata': {'name': 'e2e-models-exempt-test-...ok-opt-125m-simulated', 'namespace': 'llm'}], 'subjects': {'users': [], 'groups': [{'name': 'system:authenticated'}]}}} def _apply_cr(cr_dict): result = _run_oc(["oc", "apply", "-f", "-"], input_text=json.dumps(cr_dict)) if result.returncode != 0: meta = cr_dict.get("metadata", {}) > raise RuntimeError( f"`oc apply` failed for {cr_dict.get('kind')} " f"{meta.get('namespace', '')}/{meta.get('name')}: " f"{result.stderr.strip() or result.stdout.strip()}" ) E RuntimeError: `oc apply` failed for MaaSAuthPolicy models-as-a-service/e2e-models-exempt-test-auth: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" test/maas-e2e/test/e2e/tests/test_helper.py:767: RuntimeError ------------------------------ Captured log call ------------------------------- INFO test_helper:test_helper.py:1101 Creating MaaSAuthPolicy: e2e-models-exempt-test-auth WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 1), retrying in 2s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 2), retrying in 4s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 3), retrying in 6s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 4), retrying in 8s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 5), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 6), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 7), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 8), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 9), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 10), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 11), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 12), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 13), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 14), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 15), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 16), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 17), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 18), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" WARNING test_helper:test_helper.py:842 `oc apply -f` hit a transient error (attempt 19), retrying in 10s: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": no endpoints available for service "maas-controller-webhook-service" INFO test_subscription:test_subscription.py:848 Cleaned up models endpoint exemption test resources --- generated xml file: /workspace/artifacts-dir/e2e-system:admin-serial.xml --- - Generated html report: file:///workspace/artifacts-dir/e2e-system%3Aadmin-serial.html - =========================== short test summary info ============================ FAILED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_reject_key_for_unreconciled_subscription FAILED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAuthPolicyRemoval::test_authpolicy_deletion_revokes_access FAILED test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_auth_pass_no_subscription_gets_403 FAILED test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_rate_limit_exhaustion_gets_429 FAILED test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_models_endpoint_exempt_from_rate_limiting !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 5 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! ===== 5 failed, 1 passed, 208 deselected, 4 warnings in 753.57s (0:12:33) ====== ERROR: E2E tests failed (parallel_rc=1, serial_rc=1) ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Saved to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs Saved 9 resource type(s) to /workspace/artifacts-dir/maas-crs Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Collecting pod logs from namespace odh-ai-gateway-infra to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Saved 2 pod log file(s) to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Collecting pod logs from namespace models-as-a-service to /workspace/artifacts-dir/pod-logs/models-as-a-service Saved 0 0 pod log file(s) to /workspace/artifacts-dir/pod-logs/models-as-a-service Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Collecting pod logs from namespace kuadrant-system to /workspace/artifacts-dir/pod-logs/kuadrant-system Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/kuadrant-system Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 10 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Collecting pod logs from namespace llm to /workspace/artifacts-dir/pod-logs/llm Saved 8 pod log file(s) to /workspace/artifacts-dir/pod-logs/llm Skipping namespace istio-system (not found) ============================================== ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Saved to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs Saved 10 resource type(s) to /workspace/artifacts-dir/maas-crs Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Collecting pod logs from namespace odh-ai-gateway-infra to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Saved 2 pod log file(s) to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Collecting pod logs from namespace models-as-a-service to /workspace/artifacts-dir/pod-logs/models-as-a-service Saved 0 0 pod log file(s) to /workspace/artifacts-dir/pod-logs/models-as-a-service Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Collecting pod logs from namespace kuadrant-system to /workspace/artifacts-dir/pod-logs/kuadrant-system Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/kuadrant-system Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 10 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Collecting pod logs from namespace llm to /workspace/artifacts-dir/pod-logs/llm Saved 8 pod log file(s) to /workspace/artifacts-dir/pod-logs/llm Skipping namespace istio-system (not found) ============================================== Collecting AIGC cluster CR extensions to /workspace/artifacts-dir/maas-crs Saved datascienceclusters.yaml Saved httproutes.yaml Saved gateways.yaml Collecting MaaS diagnostics to /workspace/artifacts-dir/gather-maas (log: /workspace/artifacts-dir/maas-must-gather.log) ... MaaS must-gather complete (see /workspace/artifacts-dir/maas-must-gather.log) ======================================== Cluster / Namespace Info ======================================== --- Current context --- admin --- Logged-in user --- system:admin --- Cluster domain --- apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev DEPLOYMENT_NAMESPACE: opendatahub MAAS_API_DEPLOYMENT_NAMESPACE: odh-ai-gateway-infra MAAS_SUBSCRIPTION_NAMESPACE: models-as-a-service AUTHORINO_NAMESPACE: kuadrant-system ======================================== MaaS API Deployment ======================================== --- maas-api pods --- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES maas-api-6659bff578-bmm65 1/1 Running 0 41m 10.134.0.31 ip-10-0-134-201.us-west-2.compute.internal --- maas-api service --- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR maas-api ClusterIP 172.31.241.129 8443/TCP 41m app.kubernetes.io/component=api,app.kubernetes.io/name=maas-api,app.kubernetes.io/part-of=models-as-a-service,maas.opendatahub.io/tenant-instance=maas-api ======================================== maas-controller ======================================== --- maas-controller pods --- --- maas-controller namespace env --- INFRA_NAMESPACE=AUTO ======================================== Kuadrant Policies ======================================== --- AuthPolicies (all namespaces) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE openshift-ingress maas-gateway-auth True True Gateway maas-default-gateway 42m openshift-ingress openshift-ai-inference-authn True False Gateway openshift-ai-inference 41m --- TokenRateLimitPolicies (all namespaces) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE llm maas-trlp-facebook-opt-125m-simulated True True HTTPRoute facebook-opt-125m-simulated-kserve-route 39m llm maas-trlp-premium-simulated-simulated-premium True True HTTPRoute premium-simulated-simulated-premium-kserve-route 38m openshift-ingress gateway-default-deny True False Gateway maas-default-gateway 41m ======================================== MaaS CRs (maas.opendatahub.io) ======================================== --- Configs (cluster-scoped) --- NAME READY AGE default True 42m --- AITenants (all namespaces) --- NAMESPACE NAME READY TENANT NAMESPACE GATEWAY AGE ai-tenants models-as-a-service True models-as-a-service maas-default-gateway 41m --- Tenants (all namespaces) --- --- MaaSAuthPolicies --- NAME PHASE AGE AUTHPOLICIES e2e-apikey-unreconciled-auth 13m premium-simulator-access Active 39m simulator-access Active 39m --- MaaSSubscriptions --- NAME PHASE PRIORITY AGE e2e-apikey-unreconciled-sub 0 13m premium-simulator-subscription Active 20 39m simulator-subscription Active 10 39m --- MaaSSubscription status details --- e2e-apikey-unreconciled-sub: - premium-simulator-subscription: Active - successfully reconciled simulator-subscription: Active - successfully reconciled --- MaaSModelRefs (all namespaces) --- NAMESPACE NAME PHASE ENDPOINT HTTPROUTE GATEWAY AGE e2e-models-e2e-worker-w3-61cdc1 distinct-2-w3-61cdc1 Failed distinct-2-w3-61cdc1-kserve-route e2e-worker-w3-61cdc1 27m e2e-models-e2e-worker-w3-61cdc1 distinct-w3-61cdc1 Failed distinct-w3-61cdc1-kserve-route e2e-worker-w3-61cdc1 29m e2e-models-e2e-worker-w3-61cdc1 premium-w3-61cdc1 Failed premium-w3-61cdc1-kserve-route e2e-worker-w3-61cdc1 33m e2e-models-e2e-worker-w3-61cdc1 simulator-w3-61cdc1 Failed simulator-w3-61cdc1-kserve-route e2e-worker-w3-61cdc1 35m e2e-models-e2e-worker-w6-a5cad4 distinct-2-w6-a5cad4 Pending distinct-2-w6-a5cad4-kserve-route e2e-worker-w6-a5cad4 28m e2e-models-e2e-worker-w6-a5cad4 distinct-w6-a5cad4 Pending distinct-w6-a5cad4-kserve-route e2e-worker-w6-a5cad4 30m e2e-models-e2e-worker-w6-a5cad4 premium-w6-a5cad4 Pending premium-w6-a5cad4-kserve-route e2e-worker-w6-a5cad4 34m e2e-models-e2e-worker-w6-a5cad4 simulator-w6-a5cad4 Failed simulator-w6-a5cad4-kserve-route e2e-worker-w6-a5cad4 36m llm e2e-distinct-2-simulated Pending e2e-distinct-2-simulated-kserve-route maas-default-gateway 39m llm e2e-distinct-simulated Pending e2e-distinct-simulated-kserve-route maas-default-gateway 39m llm e2e-embedding-simulated Pending e2e-embedding-simulated-kserve-route maas-default-gateway 39m llm e2e-trlp-test-simulated Pending e2e-trlp-test-simulated-kserve-route maas-default-gateway 39m llm e2e-unconfigured-facebook-opt-125m-simulated Pending e2e-unconfigured-facebook-opt-125m-simulated-kserve-route maas-default-gateway 39m llm echo-simulator Pending echo-simulator-kserve-route maas-default-gateway 39m llm facebook-opt-125m-simulated Ready https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/ facebook-opt-125m-simulated-kserve-route maas-default-gateway 39m llm premium-simulated-simulated-premium Ready https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/ premium-simulated-simulated-premium-kserve-route maas-default-gateway 39m --- MaasTenantConfigs --- NAME READY REASON AGE INFRANAMESPACE default-tenant True Reconciled 41m odh-ai-gateway-infra --- MaasTenantConfig status details --- default-tenant: True - MaaS platform manifests applied and maas-api deployment is available --- ExternalModels (maas group) --- ======================================== Inference CRs (inference.opendatahub.io) ======================================== --- ExternalModels (inference group) --- --- ExternalProviders (inference group) --- ======================================== AI Gateway CRs ======================================== --- AIGateways (cluster-scoped) --- --- LLMBatchGateways (all namespaces) --- ======================================== KServe Resources ======================================== --- LLMInferenceServices (all namespaces) --- NAMESPACE NAME URL READY REASON AGE URLS llm e2e-distinct-2-simulated https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/llm/e2e-distinct-2-simulated True 39m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-distinct-simulated https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/llm/e2e-distinct-simulated True 39m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-embedding-simulated https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/llm/e2e-embedding-simulated True 39m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-trlp-test-simulated https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/llm/e2e-trlp-test-simulated True 39m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-unconfigured-facebook-opt-125m-simulated https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/llm/e2e-unconfigured-facebook-opt-125m-simulated True 39m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm echo-simulator https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/llm/echo-simulator True 39m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm facebook-opt-125m-simulated https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/llm/facebook-opt-125m-simulated True 39m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm premium-simulated-simulated-premium https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/llm/premium-simulated-simulated-premium True 39m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ ======================================== Kuadrant Policies (managed by maas-controller) ======================================== --- AuthPolicies (managed) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE openshift-ingress maas-gateway-auth True True Gateway maas-default-gateway 42m --- TokenRateLimitPolicies (managed) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE llm maas-trlp-facebook-opt-125m-simulated True True HTTPRoute facebook-opt-125m-simulated-kserve-route 39m llm maas-trlp-premium-simulated-simulated-premium True True HTTPRoute premium-simulated-simulated-premium-kserve-route 38m openshift-ingress gateway-default-deny True False Gateway maas-default-gateway 41m ======================================== Payload Processing / IPP ======================================== --- IPP Deployments (openshift-ingress) --- --- IPP Services (openshift-ingress) --- --- EnvoyFilters (openshift-ingress) --- NAME AGE data-science-authn-filter 43m kuadrant-auth-maas-default-gateway 41m kuadrant-maas-default-gateway 41m kuadrant-ratelimiting-maas-default-gateway 41m maas-default-gateway-authn-ssl 39m maas-model-access-logs 38m payload-processing 38m --- IPP ConfigMaps (openshift-ingress) --- ======================================== NetworkPolicies ======================================== --- NetworkPolicies (openshift-ingress) --- NAME POD-SELECTOR AGE data-science-gateway-allow gateway.networking.k8s.io/gateway-name=data-science-gateway 43m istiod-allow app=istiod 45m kube-auth-proxy app=kube-auth-proxy 43m maas-default-gateway-allow gateway.networking.k8s.io/gateway-name=maas-default-gateway 45m openshift-ai-inference-allow gateway.networking.k8s.io/gateway-name=openshift-ai-inference 45m openshift-ingress-deny-all 56m payload-processing maas.opendatahub.io/tenant-instance in (payload-pre-processing,payload-processing) 38m router-default ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default 56m ======================================== Test User Information ======================================== No test token available (not logged in via oc) ======================================== Subscription → Model Mapping ======================================== Subscription: e2e-apikey-unreconciled-sub Owner users: system:serviceaccount:llm:e2e-apikey-unreconciled-sa Owner groups: (none) Models: llm/facebook-opt-125m-simulated Subscription: premium-simulator-subscription Owner users: system:serviceaccount:premium-users-namespace:premium-service-account Owner groups: premium-user Models: llm/premium-simulated-simulated-premium Subscription: simulator-subscription Owner users: (none) Owner groups: system:authenticated Models: llm/facebook-opt-125m-simulated ======================================== Available Models (MaaSModelRefs) ======================================== Model Reference → Model ID / Endpoint e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1 → distinct-2-w3-61cdc1 (Failed) e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1 → distinct-w3-61cdc1 (Failed) e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1 → premium-w3-61cdc1 (Failed) e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1 → simulator-w3-61cdc1 (Failed) e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4 → distinct-2-w6-a5cad4 (Pending) e2e-models-e2e-worker-w6-a5cad4/distinct-w6-a5cad4 → distinct-w6-a5cad4 (Pending) e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4 → premium-w6-a5cad4 (Pending) e2e-models-e2e-worker-w6-a5cad4/simulator-w6-a5cad4 → simulator-w6-a5cad4 (Failed) llm/e2e-distinct-2-simulated → e2e-distinct-2-simulated (Pending) llm/e2e-distinct-simulated → e2e-distinct-simulated (Pending) llm/e2e-embedding-simulated → e2e-embedding-simulated (Pending) llm/e2e-trlp-test-simulated → e2e-trlp-test-simulated (Pending) llm/e2e-unconfigured-facebook-opt-125m-simulated → e2e-unconfigured-facebook-opt-125m-simulated (Pending) llm/echo-simulator → echo-simulator (Pending) llm/facebook-opt-125m-simulated → facebook-opt-125m-simulated (Ready) Endpoint: https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/ llm/premium-simulated-simulated-premium → premium-simulated-simulated-premium (Ready) Endpoint: https://maas.apps.d06c3e712c02ec79fd06.prod.konflux-ocp-ci.dev/ ======================================== Gateway / HTTPRoutes ======================================== --- Gateway --- NAME CLASS ADDRESS PROGRAMMED AGE maas-default-gateway openshift-default maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local True 45m --- HTTPRoutes (maas-api) --- NAME HOSTNAMES AGE maas-api-route 41m ======================================== Authorino ======================================== --- Authorino pods --- --- ======================================== Subscription Selector Endpoint Validation ======================================== Expected URL (from resolved maas-api deployment namespace): https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select (MAAS_API_DEPLOYMENT_NAMESPACE resolved to: odh-ai-gateway-infra) --- Sample AuthPolicy subscription-info configuration --- Inspecting: openshift-ingress/maas-gateway-auth Actual URL in AuthPolicy: https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select ✅ Request body includes requestedModel field Model reference: / Request body preview: { "groups": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.groups : (has(auth.identity.iss) ? (has(auth.identity.groups) && auth.identity.groups.all(g, g.matches('^[A-Za-z0-9:#._/ -]+$')) ? auth.identity.groups : []) : (has(auth.identity.user.groups) && auth.identity.user.groups.all(g, g.matches('^[A-Za-z0-9:#._/ -]+$')) ? auth.identity.user.groups : [])), "username": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.username : (has(auth.identity.preferred_username) ? auth.identity.preferred_username : (has(auth.identity.sub) ? auth.identity.sub : auth.identity.user.username)), "requestedSubscription": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.subscription : ("x-maas-subscription" in request.headers ? request.headers["x-maas-subscription"] : ""), "requestedModel": (size(request.path.split("/").filter(x, x != "")) >= 2 && request.path.split("/").filter(x, x != "")[0] != "v1" && request.path.split("/").filter(x, x != "")[0] != "maas-api" ? request.path.split("/").filter(x, x != "")[0] + "/" + request.path.split("/").filter(x, x != "")[1] : ("x-gateway-model-name" in request.headers ? request.headers["x-gateway-model-name"] : "")) --- Connectivity test (from kuadrant-system, simulates Authorino) --- curl -vsk -m 10 -X POST 'https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select' -H 'Content-Type: application/json' -d '{}' kubectl run failed or timed out ======================================== DNS Resolution Check ======================================== Resolving: maas-api.odh-ai-gateway-infra.svc.cluster.local nslookup failed ======================================== Configuration Summary ======================================== This summary helps compare local vs CI runs: AITenants (all namespaces): 1 MaaSModelRefs (all namespaces): 16 MaaSSubscriptions (models-as-a-service): 3 MaaSAuthPolicies (models-as-a-service): 3 Generated Kuadrant AuthPolicies: 1 Generated Kuadrant TokenRateLimitPolicies: 3 LLMInferenceServices (all namespaces): 8 ExternalModels - inference (all namespaces): 0 ExternalProviders - inference (all namespaces): 0 EnvoyFilters (openshift-ingress): 7 IPP Deployments (openshift-ingress): 0 NetworkPolicies (openshift-ingress): 8 Subscription selector URL: https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select Test user: system:admin