Fetching https://github.com/opendatahub-io/models-as-a-service@7879675aa66d002425c7b32375d6d0156eaa0c56 into /workspace/source/test/maas-e2e ... MaaS checkout at: 7879675aa66d002425c7b32375d6d0156eaa0c56 Patched /workspace/source/test/maas-e2e/scripts/deploy.sh Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS checkout fixture root) NOTE: rewriting AI_GATEWAY_CONTROLLER_IMAGE from snapshot digest to tag odh-pr-199 MaaS platform images: MAAS_API_IMAGE=quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 MAAS_CONTROLLER_IMAGE=quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 AI_GATEWAY_CONTROLLER_IMAGE=quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-199 PRAXIS_EXTPROC_IMAGE=quay.io/opendatahub/odh-praxis-extproc:odh-stable ---------------------------------------- ai-gateway-controller E2E on OpenShift ---------------------------------------- Checking prerequisites... Prerequisites met — logged in as: system:admin DEPLOY_MODE: kustomize MAAS_API_IMAGE: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 MAAS_CONTROLLER_IMAGE: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 AI_GATEWAY_CONTROLLER_IMAGE: quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-199 PRAXIS_EXTPROC_IMAGE: quay.io/opendatahub/odh-praxis-extproc:odh-stable 2026-10-30T13:59:56Z deploy_platform start Deploying MaaS platform via ODH operator... Gateway ingress mode for deploy.sh: ocproute (loadbalancer | ocproute) Using custom MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 Using custom MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 Deployment mode: kustomize Using policy engine: rhcl (Authorino namespace: kuadrant-system) [INFO] =================================================== [INFO] Models-as-a-Service Deployment [INFO] =================================================== [INFO] Validating configuration... [INFO] Configuration validated successfully [INFO] Deployment configuration: [INFO] Mode: kustomize [INFO] Operator: odh [INFO] Policy Engine: rhcl [INFO] Namespace: opendatahub [INFO] TLS Backend: true [INFO] External OIDC: false [INFO] MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] Installing shared dependencies for kustomize mode... [INFO] =================================================== [INFO] Setup Shared Dependencies (Helm) [INFO] =================================================== [INFO] Operator type: odh [INFO] Deployment mode: kustomize [INFO] Ingress mode: ocproute [INFO] Policy engine: rhcl [INFO] Chart source: https://github.com/opendatahub-io/odh-gitops.git @ main (fadfb31e89ae898b04a9a75bfa8bb91588d7e625) [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] Payload processing image: chart default [INFO] AI Gateway operator image: catalog default [INFO] helm not found, installing v3.17.3... [INFO] helm installed: v3.17.3+ge4da497 [INFO] Cloning odh-gitops chart (branch: main, commit: fadfb31e89ae898b04a9a75bfa8bb91588d7e625)... Cloning into '/tmp/tmp.R6fUcQaG7Z/odh-gitops'... From https://github.com/opendatahub-io/odh-gitops * branch fadfb31e89ae898b04a9a75bfa8bb91588d7e625 -> FETCH_HEAD [INFO] Chart resolved at: /tmp/tmp.R6fUcQaG7Z/odh-gitops/charts/rhai-on-openshift-chart [INFO] [INFO] Phase 1: Installing operators via Helm chart... [INFO] Running: helm upgrade --install rhoai-deps Release "rhoai-deps" does not exist. Installing it now. NAME: rhoai-deps LAST DEPLOYED: Fri Oct 30 13:59:58 2026 NAMESPACE: rhoai-deps STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: ODH Chart installed successfully! === Installation Summary === Operator: ODH Namespace: rhoai-deps Profile: maas === Enabled Components === - aigateway: Removed - aipipelines: Removed - dashboard: Removed - feastoperator: Removed - kserve: Managed - kueue: Removed - mlflowoperator: Removed - modelregistry: Removed - ogx: Removed - ray: Removed - sparkoperator: Removed - trainer: Removed - trainingoperator: Removed - trustyai: Removed - workbenches: Removed === Enabled Services === - monitoring: Removed === Enabled Dependencies === - certManager - jobSet - rhcl === Next Steps === 1. Wait for operators to be ready: kubectl get csv -A | grep -E "(rhods|opendatahub|cert-manager|leader-worker)" 2. If CRs were not created (first install), run the same helm upgrade command again. 3. Check DataScienceCluster status: kubectl get datasciencecluster default-dsc -o jsonpath='{.status.phase}' === Enable Authorino TLS (Optional) === To enable TLS for Authorino: 1. Annotate the service to trigger TLS certificate generation: kubectl annotate svc/authorino-authorino-authorization \ service.beta.openshift.io/serving-cert-secret-name=authorino-server-cert \ -n kuadrant-system 2. Patch the Authorino CR to enable TLS: kubectl patch authorino/authorino -n kuadrant-system --type=merge -p '{"spec":{"clusterWide":true,"listener":{"tls":{"enabled":true,"certSecretRef":{"name":"authorino-server-cert"}}},"oidcServer":{"tls":{"enabled":false}}}}' === Documentation === For more information, see the chart README: https://github.com/opendatahub-io/odh-gitops/tree/main/charts/rhai-on-openshift-chart [INFO] [INFO] Waiting for operator CRDs to be established... ⏳ Waiting for CRD dscinitializations.dscinitialization.opendatahub.io to appear (timeout: 600s)… ✅ CRD dscinitializations.dscinitialization.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io condition met ⏳ Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 600s)… ✅ CRD datascienceclusters.datasciencecluster.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met ⏳ Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 600s)… ✅ CRD kuadrants.kuadrant.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [INFO] [INFO] Waiting for ODH operator to be ready (webhook must be serving)... [INFO] Operator CSV phase: Installing (0s / 300s) [INFO] Operator CSV is Succeeded in opendatahub-operator-system deployment.apps/opendatahub-operator-controller-manager condition met [INFO] [INFO] Applying local MaaS CRDs for kustomize mode... ⏳ Applying MaaS Controller CRDs from /workspace/source/test/maas-e2e/deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io created ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] [INFO] Phase 2: Applying CRD-dependent resources (DSC, DSCI, Kuadrant CR)... [INFO] Running: helm upgrade --install rhoai-deps Release "rhoai-deps" has been upgraded. Happy Helming! NAME: rhoai-deps LAST DEPLOYED: Fri Oct 30 14:01:03 2026 NAMESPACE: rhoai-deps STATUS: deployed REVISION: 2 TEST SUITE: None NOTES: ODH Chart installed successfully! === Installation Summary === Operator: ODH Namespace: rhoai-deps Profile: maas === Enabled Components === - aigateway: Removed - aipipelines: Removed - dashboard: Removed - feastoperator: Removed - kserve: Managed - kueue: Removed - mlflowoperator: Removed - modelregistry: Removed - ogx: Removed - ray: Removed - sparkoperator: Removed - trainer: Removed - trainingoperator: Removed - trustyai: Removed - workbenches: Removed === Enabled Services === - monitoring: Removed === Enabled Dependencies === - certManager - jobSet - rhcl === Next Steps === 1. Wait for operators to be ready: kubectl get csv -A | grep -E "(rhods|opendatahub|cert-manager|leader-worker)" 2. If CRs were not created (first install), run the same helm upgrade command again. 3. Check DataScienceCluster status: kubectl get datasciencecluster default-dsc -o jsonpath='{.status.phase}' === Enable Authorino TLS (Optional) === To enable TLS for Authorino: 1. Annotate the service to trigger TLS certificate generation: kubectl annotate svc/authorino-authorino-authorization \ service.beta.openshift.io/serving-cert-secret-name=authorino-server-cert \ -n kuadrant-system 2. Patch the Authorino CR to enable TLS: kubectl patch authorino/authorino -n kuadrant-system --type=merge -p '{"spec":{"clusterWide":true,"listener":{"tls":{"enabled":true,"certSecretRef":{"name":"authorino-server-cert"}}},"oidcServer":{"tls":{"enabled":false}}}}' === Documentation === For more information, see the chart README: https://github.com/opendatahub-io/odh-gitops/tree/main/charts/rhai-on-openshift-chart [INFO] Waiting for Kuadrant policy engine to become ready... [INFO] Waiting for: Kuadrant ready in kuadrant-system (timeout: 600s) [INFO] Kuadrant ready in kuadrant-system - Ready [INFO] [INFO] Shared dependencies installed successfully [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [WARN] DEPLOYING POC POSTGRESQL — NOT INTENDED FOR PRODUCTION USE [WARN] Data is stored in ephemeral storage and will be lost on pod restart. [WARN] For production, use --postgres-connection with an external database [WARN] (AWS RDS, Crunchy Operator, Azure Database, etc.) [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ⚠️ WARNING FOR PRODUCTION USE. ⚠️ ┃ ┃ This deploys PostgreSQL with ephemeral storage (emptyDir). ┃ ┃ Data WILL be lost on pod restart. ┃ ┃ For production, use an external database: ┃ ┃ deploy.sh --postgres-connection postgresql://... ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 🔧 Fresh install: Deploying PostgreSQL in infrastructure namespace 'odh-ai-gateway-infra'... 📦 Creating infrastructure namespace 'odh-ai-gateway-infra'... namespace/odh-ai-gateway-infra created Generated random PostgreSQL password (stored in secret postgres-creds) Creating PostgreSQL deployment... ⚠️ Using POC configuration (ephemeral storage) Using default PostgreSQL image (operator CSV not available) Image: registry.redhat.io/rhel9/postgresql-16:latest secret/postgres-creds created deployment.apps/postgres created service/postgres created secret/maas-db-config created Waiting for PostgreSQL to be ready... deployment.apps/postgres condition met ✅ PostgreSQL deployed successfully Namespace: odh-ai-gateway-infra Database: maas User: maas Secret: maas-db-config (contains DB_CONNECTION_URL with FQDN) ⚠️ For production, use AWS RDS, Crunchy Operator, or Azure Database Note: Schema migrations run automatically when maas-api starts [INFO] Shared dependencies installed [INFO] [INFO] MaaS Controller... [INFO] Phase 1: Applying MaaS CRDs and waiting until Established (controller creates Config after CRD is ready)... ⏳ Applying MaaS Controller CRDs from /workspace/source/test/maas-e2e/scripts/../deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io unchanged ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] Phase 2: Applying full controller kustomize (same as operator: deployment/base/maas-controller/default)... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io configured serviceaccount/maas-controller created role.rbac.authorization.k8s.io/maas-controller-leader-election-role created clusterrole.rbac.authorization.k8s.io/maas-controller-cluster-config-role created clusterrole.rbac.authorization.k8s.io/maas-controller-ocp-role created clusterrole.rbac.authorization.k8s.io/maas-controller-role created clusterrole.rbac.authorization.k8s.io/maas-owner-role created clusterrole.rbac.authorization.k8s.io/maas-viewer-role created rolebinding.rbac.authorization.k8s.io/maas-controller-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-cluster-config-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-ocp-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-rolebinding created configmap/maas-parameters created service/maas-controller-metrics created service/maas-controller-webhook-service created deployment.apps/maas-controller created servicemonitor.monitoring.coreos.com/maas-controller-metrics created networkpolicy.networking.k8s.io/maas-controller-allow-monitoring created validatingwebhookconfiguration.admissionregistration.k8s.io/maas-validating-webhook-configuration created [INFO] Restarting maas-controller to pick up configuration changes... deployment.apps/maas-controller restarted [INFO] Waiting for maas-controller to be ready... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... deployment "maas-controller" successfully rolled out [INFO] Controller ready. [INFO] [INFO] Waiting for Tenant reconciler to deploy maas-api... [INFO] Applying secret migration RBAC to namespace odh-ai-gateway-infra... role.rbac.authorization.k8s.io/maas-controller-secret-migrate created rolebinding.rbac.authorization.k8s.io/maas-controller-secret-migrate created [INFO] maas-api deployment found in odh-ai-gateway-infra, waiting for rollout... Waiting for deployment "maas-api" rollout to finish: 0 of 1 updated replicas are available... deployment "maas-api" successfully rolled out [INFO] maas-api is ready [INFO] Configuring TLS backend for Authorino and MaaS API... * Waiting for deployment/authorino in kuadrant-system (timeout: 300s)... * Found deployment/authorino [INFO] Running TLS configuration script... [INFO] TLS configuration script completed successfully [INFO] Restarting deployments to pick up TLS configuration... deployment.apps/maas-api restarted deployment.apps/authorino restarted [INFO] Waiting for Authorino deployment to be ready... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out [INFO] TLS backend configuration complete [INFO] [INFO] MaaS API and MaaS Controller deployment completed successfully! [INFO] maas-api image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 (namespace: odh-ai-gateway-infra) [INFO] maas-controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 (namespace: opendatahub) [INFO] =================================================== [INFO] Models-as-a-Service Deployment completed successfully! [INFO] =================================================== * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' Raising llmisvc-controller-manager CPU in opendatahub (request 1, limit 2)... deployment.apps/llmisvc-controller-manager resource requirements updated Waiting for deployment "llmisvc-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "llmisvc-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "llmisvc-controller-manager" rollout to finish: 1 old replicas are pending termination... deployment "llmisvc-controller-manager" successfully rolled out llmisvc-controller-manager resources: {"limits":{"cpu":"2","memory":"300Mi"},"requests":{"cpu":"1","memory":"300Mi"}} ⚠️ WARNING: Skipping Authorino readiness check (SKIP_AUTH_CHECK=true) ✅ MaaS platform deployment completed 2026-10-30T14:03:46Z deploy_platform end ---------------------------------------- Deploying Models ---------------------------------------- 2026-10-30T14:03:46Z deploy_models start Deploying MaaS system (free + premium: LLMIS + MaaSModelRef + MaaSAuthPolicy + MaaSSubscription) Waiting for Gateway openshift-ingress/maas-default-gateway to be Programmed=True (timeout: 600s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met ✅ Gateway openshift-ingress/maas-default-gateway is Programmed Creating 'llm' namespace... namespace/llm created 'models-as-a-service' namespace already exists Waiting for ready endpoints on maas-controller-webhook-service in opendatahub (timeout: 120s)... ✅ MaaS admission webhook has ready endpoints: 10.132.0.44 maasauthpolicy.maas.opendatahub.io/premium-simulator-access created maasauthpolicy.maas.opendatahub.io/simulator-access created maasmodelref.maas.opendatahub.io/e2e-distinct-2-simulated created maasmodelref.maas.opendatahub.io/e2e-distinct-simulated created maasmodelref.maas.opendatahub.io/e2e-embedding-simulated created maasmodelref.maas.opendatahub.io/e2e-trlp-test-simulated created maasmodelref.maas.opendatahub.io/e2e-unconfigured-facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/echo-simulator created maasmodelref.maas.opendatahub.io/facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/premium-simulated-simulated-premium created maassubscription.maas.opendatahub.io/premium-simulator-subscription created maassubscription.maas.opendatahub.io/simulator-subscription created llminferenceservice.serving.kserve.io/e2e-distinct-2-simulated created llminferenceservice.serving.kserve.io/e2e-distinct-simulated created llminferenceservice.serving.kserve.io/e2e-embedding-simulated created llminferenceservice.serving.kserve.io/e2e-trlp-test-simulated created llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/echo-simulator created llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium created ✅ MaaS system deployed (free + premium + e2e test fixtures) Waiting for models to be ready (timeout: 300s)... llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated condition met llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium condition met llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated condition met ✅ Simulator models ready Waiting for governed MaaSModelRefs to be Ready (timeout: 300s per model)... maasmodelref.maas.opendatahub.io/facebook-opt-125m-simulated condition met maasmodelref.maas.opendatahub.io/premium-simulated-simulated-premium condition met ✅ Governed MaaSModelRefs ready Waiting for AuthPolicies targeting gateway 'maas-default-gateway' to be enforced (timeout: 180s)... Skipping openshift-ai-inference-authn (targets openshift-ai-inference, not maas-default-gateway) ✅ All AuthPolicies targeting maas-default-gateway enforced (1 policies) 2026-10-30T14:04:31Z deploy_models end Patching Authorino to log_level DEBUG... authorino.operator.authorino.kuadrant.io/authorino patched ✅ Authorino patched to log_level DEBUG deployment "authorino" successfully rolled out ---------------------------------------- Deploying ai-gateway-controller ---------------------------------------- 2026-10-30T14:04:32Z deploy_ai_gateway_controller start Deploying ai-gateway-controller image: quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-199 praxis-extproc image: quay.io/opendatahub/odh-praxis-extproc:odh-stable namespace: opendatahub gateway: openshift-ingress/maas-default-gateway customresourcedefinition.apiextensions.k8s.io/aiguardrails.aigateway.opendatahub.io created serviceaccount/ai-gateway-controller created clusterrole.rbac.authorization.k8s.io/ai-gateway-controller-role created clusterrolebinding.rbac.authorization.k8s.io/ai-gateway-controller-rolebinding created configmap/ai-gateway-controller-parameters created service/ai-gateway-controller-metrics created deployment.apps/ai-gateway-controller created servicemonitor.monitoring.coreos.com/ai-gateway-controller-metrics created networkpolicy.networking.k8s.io/ai-gateway-controller-allow-metrics-traffic created Waiting for deployment "ai-gateway-controller" rollout to finish: 0 of 1 updated replicas are available... deployment "ai-gateway-controller" successfully rolled out Triggering immediate praxis-extproc reconcile ... deployment.apps/ai-gateway-controller restarted Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... deployment "ai-gateway-controller" successfully rolled out Waiting for shared praxis-extproc (quay.io/opendatahub/odh-praxis-extproc:odh-stable) in openshift-ingress: payload-processing and payload-pre-processing (timeout: 300s) ... ✅ shared praxis-extproc ready: openshift-ingress/payload-processing and openshift-ingress/payload-pre-processing Annotating praxis IPP resources opendatahub.io/managed=false so maas-controller skips them ... deployment.apps/payload-processing annotated service/payload-processing annotated destinationrule.networking.istio.io/payload-processing annotated deployment.apps/payload-pre-processing annotated service/payload-pre-processing annotated destinationrule.networking.istio.io/payload-pre-processing annotated envoyfilter.networking.istio.io/payload-processing annotated networkpolicy.networking.k8s.io/payload-processing annotated Verified shared ext_proc dataplane image: openshift-ingress/payload-processing=quay.io/opendatahub/odh-praxis-extproc:odh-stable ai-gateway-controller rollout complete 2026-10-30T14:05:03Z deploy_ai_gateway_controller end ---------------------------------------- Enabling tenant namespace discovery ---------------------------------------- 2026-10-30T14:05:03Z tenant_namespace_discovery start Enabling --enable-tenant-namespace-discovery on maas-controller... maas-controller already has tenant namespace discovery enabled 2026-10-30T14:05:03Z tenant_namespace_discovery end ---------------------------------------- Setting up variables for tests ---------------------------------------- -- Setting up variables for tests -- HOST: maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev MAAS_API_BASE_URL: https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/maas-api ---------------------------------------- Setting up test tokens ---------------------------------------- Setting up premium test token (SA-based, works when oc whoami -t is unavailable)... Creating namespace: premium-users-namespace namespace/premium-users-namespace created Creating service account: premium-service-account serviceaccount/premium-service-account created Patching MaaSAuthPolicy premium-simulator-access to include system:serviceaccount:premium-users-namespace:premium-service-account... maasauthpolicy.maas.opendatahub.io/premium-simulator-access patched Patching MaaSSubscription premium-simulator-subscription to include system:serviceaccount:premium-users-namespace:premium-service-account... maassubscription.maas.opendatahub.io/premium-simulator-subscription patched Waiting for MaaSSubscriptions to reconcile after patch (timeout: 60s)... ✅ Both subscriptions ready: simulator-subscription=Active, premium-simulator-subscription=Active ✅ Premium test token setup complete (E2E_TEST_TOKEN_SA_* exported) Setting up test tokens (admin + regular user)... Current admin session: system:admin (will be preserved) ⚠️ No htpasswd token available - using SA token (admin tests may fail) Creating namespace: maas-admin namespace/maas-admin created Creating service account: tester-admin-user in maas-admin serviceaccount/tester-admin-user created Creating cluster role binding for tester-admin-user clusterrolebinding.rbac.authorization.k8s.io/tester-admin-user-binding created ✅ User setup completed: tester-admin-user (namespace: maas-admin) role.rbac.authorization.k8s.io/maas-admin-e2e created rolebinding.rbac.authorization.k8s.io/maas-admin-e2e-system-serviceaccount-maas-admin-tester-admin-user created auth.services.platform.opendatahub.io/auth patched ✅ Added system:serviceaccounts:maas-admin to Auth CR adminGroups (SA admin fallback) clusterrole.rbac.authorization.k8s.io/maas-admin created rolebinding.rbac.authorization.k8s.io/odh-admins-maas-admin created Creating separate SA token for regular user (required for IDOR tests)... Creating service account: tester-regular-user in default serviceaccount/tester-regular-user created Creating cluster role binding for tester-regular-user clusterrolebinding.rbac.authorization.k8s.io/tester-regular-user-binding created ✅ User setup completed: tester-regular-user (namespace: default) ✅ Regular user token for tester-regular-user (SA-based, namespace: default) Token setup complete (main session unchanged: system:admin) ---------------------------------------- Ensuring gateway allows model HTTPRoutes ---------------------------------------- /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 112: CUSTOM_RESOURCE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 113: NAMESPACE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 114: RESOURCE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 115: CRD_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 116: CSV_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 117: SUBSCRIPTION_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 118: POD_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 119: WEBHOOK_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 120: CUSTOM_CHECK_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 121: AUTHORINO_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 122: ROLLOUT_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 123: KUBECONFIG_WAIT_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 124: CATALOGSOURCE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 125: LLMIS_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 126: MAASMODELREF_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 127: AUTHPOLICY_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 130: _MAX_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 165: LOG_LEVEL_DEBUG: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 166: LOG_LEVEL_INFO: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 167: LOG_LEVEL_WARN: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 168: LOG_LEVEL_ERROR: readonly variable ./test/e2e/scripts/prow_run_ai_gateway_controller_test.sh: line 133: derive_infra_namespace: command not found Patching gateway allowedRoutes to include llm (was: ) ... [INFO] Updating Gateway allowedRoutes on 1 listener(s)... gateway.gateway.networking.k8s.io/maas-default-gateway patched ---------------------------------------- Validating Deployment ---------------------------------------- 2026-10-30T14:05:09Z validate start Deployment Validation ========================================= 🚀 MaaS Platform Deployment Validation ========================================= ========================================= 1️⃣ Component Status Checks ========================================= 🔍 Checking: MaaS API pods ✅ PASS: MaaS API has 1 running pod(s) 🔍 Checking: Policy engine pods (RHCL/Kuadrant) ✅ PASS: Policy engine has 7 running pod(s) in kuadrant-system 🔍 Checking: OpenDataHub/KServe pods ℹ️ opendatahub namespace: 8 running pod(s) ✅ PASS: OpenDataHub/RHOAI has 8 total running pod(s) 🔍 Checking: LLM namespace and models ✅ PASS: Found 8 LLMInferenceService(s) with 8 running pod(s) ========================================= 2️⃣ Gateway Status ========================================= 🔍 Checking: Gateway resource ✅ PASS: Gateway is Accepted and Programmed 🔍 Checking: HTTPRoute for maas-api ⚠️ WARNING: HTTPRoute maas-api-route exists but acceptance status unclear  Note: This is usually fine if other checks pass 🔍 Checking: Gateway hostname ✅ PASS: Gateway hostname (from MAAS_GATEWAY_HOST): https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev ========================================= 3️⃣ Policy Status ========================================= 🔍 Checking: AuthPolicy ✅ PASS: AuthPolicy is configured and accepted 🔍 Checking: TokenRateLimitPolicy ✅ PASS: TokenRateLimitPolicy is configured and accepted ========================================= 4️⃣ API Endpoint Tests ========================================= ℹ️ Using gateway endpoint: https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev 🔍 Checking: Authentication token ✅ PASS: OpenShift identity token available 🔍 Checking: MaaS API key creation ❌ FAIL: Failed to create MaaS API key (HTTP 404)  Reason: Response:   Suggestion: Check MaaS API key endpoint: https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/maas-api/v1/api-keys 🔍 Checking: Models endpoint ⚠️ WARNING: Skipping models endpoint test  Note: No authentication token available 🔍 Checking: Authorization enforcement (401 without token) ========================================= 📊 Validation Summary ========================================= Results: ✅ Passed: 9 ❌ Failed: 1 ⚠️ Warnings: 2 ❌ FAIL: Some checks failed. Please review the errors above. Common fixes: - Wait for pods to start: kubectl get pods -A | grep -v Running - Check operator logs: kubectl logs -n kuadrant-system -l app.kubernetes.io/name=kuadrant-operator - Re-run deployment: ./scripts/deploy.sh Usage: ./scripts/validate-deployment.sh [MODEL_NAME] MODEL_NAME: Optional. Specify a model to validate against First validation failed; retrying after short wait... Waiting for Gateway openshift-ingress/maas-default-gateway to be Programmed=True (timeout: 60s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met ✅ Gateway openshift-ingress/maas-default-gateway is Programmed deployment.apps/maas-controller condition met deployment.apps/maas-api condition met ========================================= 🚀 MaaS Platform Deployment Validation ========================================= ========================================= 1️⃣ Component Status Checks ========================================= 🔍 Checking: MaaS API pods ✅ PASS: MaaS API has 1 running pod(s) 🔍 Checking: Policy engine pods (RHCL/Kuadrant) ✅ PASS: Policy engine has 7 running pod(s) in kuadrant-system 🔍 Checking: OpenDataHub/KServe pods ℹ️ opendatahub namespace: 8 running pod(s) ✅ PASS: OpenDataHub/RHOAI has 8 total running pod(s) 🔍 Checking: LLM namespace and models ✅ PASS: Found 8 LLMInferenceService(s) with 8 running pod(s) ========================================= 2️⃣ Gateway Status ========================================= 🔍 Checking: Gateway resource ✅ PASS: Gateway is Accepted and Programmed 🔍 Checking: HTTPRoute for maas-api ⚠️ WARNING: HTTPRoute maas-api-route exists but acceptance status unclear  Note: This is usually fine if other checks pass 🔍 Checking: Gateway hostname ✅ PASS: Gateway hostname (from MAAS_GATEWAY_HOST): https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev ========================================= 3️⃣ Policy Status ========================================= 🔍 Checking: AuthPolicy ✅ PASS: AuthPolicy is configured and accepted 🔍 Checking: TokenRateLimitPolicy ✅ PASS: TokenRateLimitPolicy is configured and accepted ========================================= 4️⃣ API Endpoint Tests ========================================= ℹ️ Using gateway endpoint: https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev 🔍 Checking: Authentication token ✅ PASS: OpenShift identity token available 🔍 Checking: MaaS API key creation ❌ FAIL: Failed to create MaaS API key (HTTP 404)  Reason: Response:   Suggestion: Check MaaS API key endpoint: https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/maas-api/v1/api-keys 🔍 Checking: Models endpoint ⚠️ WARNING: Skipping models endpoint test  Note: No authentication token available 🔍 Checking: Authorization enforcement (401 without token) ========================================= 📊 Validation Summary ========================================= Results: ✅ Passed: 9 ❌ Failed: 1 ⚠️ Warnings: 2 ❌ FAIL: Some checks failed. Please review the errors above. Common fixes: - Wait for pods to start: kubectl get pods -A | grep -v Running - Check operator logs: kubectl logs -n kuadrant-system -l app.kubernetes.io/name=kuadrant-operator - Re-run deployment: ./scripts/deploy.sh Usage: ./scripts/validate-deployment.sh [MODEL_NAME] MODEL_NAME: Optional. Specify a model to validate against ERROR: Deployment validation failed after retry ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Saved to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs Saved 9 resource type(s) to /workspace/artifacts-dir/maas-crs Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 8 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Collecting pod logs from namespace odh-ai-gateway-infra to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Saved 2 pod log file(s) to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Collecting pod logs from namespace models-as-a-service to /workspace/artifacts-dir/pod-logs/models-as-a-service Saved 0 0 pod log file(s) to /workspace/artifacts-dir/pod-logs/models-as-a-service Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Collecting pod logs from namespace kuadrant-system to /workspace/artifacts-dir/pod-logs/kuadrant-system Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/kuadrant-system Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 15 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Collecting pod logs from namespace llm to /workspace/artifacts-dir/pod-logs/llm Saved 8 pod log file(s) to /workspace/artifacts-dir/pod-logs/llm Skipping namespace istio-system (not found) ============================================== ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Saved to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs Saved 10 resource type(s) to /workspace/artifacts-dir/maas-crs Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 8 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Collecting pod logs from namespace odh-ai-gateway-infra to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Saved 2 pod log file(s) to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Collecting pod logs from namespace models-as-a-service to /workspace/artifacts-dir/pod-logs/models-as-a-service Saved 0 0 pod log file(s) to /workspace/artifacts-dir/pod-logs/models-as-a-service Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Collecting pod logs from namespace kuadrant-system to /workspace/artifacts-dir/pod-logs/kuadrant-system Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/kuadrant-system Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 15 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Collecting pod logs from namespace llm to /workspace/artifacts-dir/pod-logs/llm Saved 8 pod log file(s) to /workspace/artifacts-dir/pod-logs/llm Skipping namespace istio-system (not found) ============================================== Collecting AIGC cluster CR extensions to /workspace/artifacts-dir/maas-crs Saved datascienceclusters.yaml Saved httproutes.yaml Saved gateways.yaml Collecting MaaS diagnostics to /workspace/artifacts-dir/gather-maas (log: /workspace/artifacts-dir/maas-must-gather.log) ... MaaS must-gather complete (see /workspace/artifacts-dir/maas-must-gather.log) ======================================== Cluster / Namespace Info ======================================== --- Current context --- admin --- Logged-in user --- system:admin --- Cluster domain --- apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev DEPLOYMENT_NAMESPACE: opendatahub MAAS_API_DEPLOYMENT_NAMESPACE: odh-ai-gateway-infra MAAS_SUBSCRIPTION_NAMESPACE: models-as-a-service AUTHORINO_NAMESPACE: kuadrant-system ======================================== MaaS API Deployment ======================================== --- maas-api pods --- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES maas-api-b55fcb7f8-rvd8w 1/1 Running 0 3m13s 10.134.0.27 ip-10-0-140-44.ec2.internal --- maas-api service --- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR maas-api ClusterIP 172.31.100.196 8443/TCP 3m24s app.kubernetes.io/component=api,app.kubernetes.io/name=maas-api,app.kubernetes.io/part-of=models-as-a-service,maas.opendatahub.io/tenant-instance=maas-api ======================================== maas-controller ======================================== --- maas-controller pods --- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES maas-controller-797b999b7c-rw6fh 1/1 Running 0 3m58s 10.132.0.44 ip-10-0-142-58.ec2.internal --- maas-controller namespace env --- INFRA_NAMESPACE=AUTO ======================================== Kuadrant Policies ======================================== --- AuthPolicies (all namespaces) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE openshift-ingress maas-gateway-auth True True Gateway maas-default-gateway 3m55s openshift-ingress openshift-ai-inference-authn True False Gateway openshift-ai-inference 4m11s --- TokenRateLimitPolicies (all namespaces) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE llm maas-trlp-facebook-opt-125m-simulated True True HTTPRoute facebook-opt-125m-simulated-kserve-route 2m10s llm maas-trlp-premium-simulated-simulated-premium True True HTTPRoute premium-simulated-simulated-premium-kserve-route 2m7s openshift-ingress gateway-default-deny True False Gateway maas-default-gateway 3m24s ======================================== MaaS CRs (maas.opendatahub.io) ======================================== --- Configs (cluster-scoped) --- NAME READY AGE default True 3m55s --- AITenants (all namespaces) --- NAMESPACE NAME READY TENANT NAMESPACE GATEWAY AGE ai-tenants models-as-a-service True models-as-a-service maas-default-gateway 3m28s --- Tenants (all namespaces) --- --- MaaSAuthPolicies --- NAME PHASE AGE AUTHPOLICIES premium-simulator-access Active 2m37s simulator-access Active 2m37s --- MaaSSubscriptions --- NAME PHASE PRIORITY AGE premium-simulator-subscription Active 20 2m36s simulator-subscription Active 10 2m36s --- MaaSSubscription status details --- premium-simulator-subscription: Active - successfully reconciled simulator-subscription: Active - successfully reconciled --- MaaSModelRefs (all namespaces) --- NAMESPACE NAME PHASE ENDPOINT HTTPROUTE GATEWAY AGE llm e2e-distinct-2-simulated Pending e2e-distinct-2-simulated-kserve-route maas-default-gateway 2m37s llm e2e-distinct-simulated Pending e2e-distinct-simulated-kserve-route maas-default-gateway 2m37s llm e2e-embedding-simulated Pending e2e-embedding-simulated-kserve-route maas-default-gateway 2m37s llm e2e-trlp-test-simulated Pending e2e-trlp-test-simulated-kserve-route maas-default-gateway 2m37s llm e2e-unconfigured-facebook-opt-125m-simulated Pending e2e-unconfigured-facebook-opt-125m-simulated-kserve-route maas-default-gateway 2m36s llm echo-simulator Pending echo-simulator-kserve-route maas-default-gateway 2m36s llm facebook-opt-125m-simulated Ready https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/ facebook-opt-125m-simulated-kserve-route maas-default-gateway 2m36s llm premium-simulated-simulated-premium Ready https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/ premium-simulated-simulated-premium-kserve-route maas-default-gateway 2m36s --- MaasTenantConfigs --- NAME READY REASON AGE INFRANAMESPACE default-tenant True Reconciled 3m26s odh-ai-gateway-infra --- MaasTenantConfig status details --- default-tenant: True - MaaS platform manifests applied and maas-api deployment is available --- ExternalModels (maas group) --- ======================================== Inference CRs (inference.opendatahub.io) ======================================== --- ExternalModels (inference group) --- --- ExternalProviders (inference group) --- ======================================== AI Gateway CRs ======================================== --- AIGateways (cluster-scoped) --- --- LLMBatchGateways (all namespaces) --- ======================================== KServe Resources ======================================== --- LLMInferenceServices (all namespaces) --- NAMESPACE NAME URL READY REASON AGE URLS llm e2e-distinct-2-simulated https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/llm/e2e-distinct-2-simulated True 2m38s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-distinct-simulated https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/llm/e2e-distinct-simulated True 2m38s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-embedding-simulated https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/llm/e2e-embedding-simulated True 2m37s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-trlp-test-simulated https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/llm/e2e-trlp-test-simulated True 2m37s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-unconfigured-facebook-opt-125m-simulated https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/llm/e2e-unconfigured-facebook-opt-125m-simulated True 2m37s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm echo-simulator https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/llm/echo-simulator True 2m37s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm facebook-opt-125m-simulated https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/llm/facebook-opt-125m-simulated True 2m37s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm premium-simulated-simulated-premium https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/llm/premium-simulated-simulated-premium True 2m36s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ ======================================== Kuadrant Policies (managed by maas-controller) ======================================== --- AuthPolicies (managed) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE openshift-ingress maas-gateway-auth True True Gateway maas-default-gateway 3m58s --- TokenRateLimitPolicies (managed) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE llm maas-trlp-facebook-opt-125m-simulated True True HTTPRoute facebook-opt-125m-simulated-kserve-route 2m13s llm maas-trlp-premium-simulated-simulated-premium True True HTTPRoute premium-simulated-simulated-premium-kserve-route 2m10s openshift-ingress gateway-default-deny True False Gateway maas-default-gateway 3m27s ======================================== Payload Processing / IPP ======================================== --- IPP Deployments (openshift-ingress) --- --- IPP Services (openshift-ingress) --- --- EnvoyFilters (openshift-ingress) --- NAME AGE data-science-authn-filter 5m21s kuadrant-auth-maas-default-gateway 3m26s kuadrant-maas-default-gateway 3m26s kuadrant-ratelimiting-maas-default-gateway 3m26s maas-default-gateway-authn-ssl 2m38s maas-model-access-logs 108s payload-processing 109s --- IPP ConfigMaps (openshift-ingress) --- ======================================== NetworkPolicies ======================================== --- NetworkPolicies (openshift-ingress) --- NAME POD-SELECTOR AGE data-science-gateway-allow gateway.networking.k8s.io/gateway-name=data-science-gateway 5m23s istiod-allow app=istiod 6m28s kube-auth-proxy app=kube-auth-proxy 5m22s maas-default-gateway-allow gateway.networking.k8s.io/gateway-name=maas-default-gateway 6m28s openshift-ai-inference-allow gateway.networking.k8s.io/gateway-name=openshift-ai-inference 6m28s openshift-ingress-deny-all 19m payload-processing maas.opendatahub.io/tenant-instance in (payload-pre-processing,payload-processing) 110s router-default ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default 19m ======================================== Test User Information ======================================== No test token available (not logged in via oc) ======================================== Subscription → Model Mapping ======================================== Subscription: premium-simulator-subscription Owner users: system:serviceaccount:premium-users-namespace:premium-service-account Owner groups: premium-user Models: llm/premium-simulated-simulated-premium Subscription: simulator-subscription Owner users: (none) Owner groups: system:authenticated Models: llm/facebook-opt-125m-simulated ======================================== Available Models (MaaSModelRefs) ======================================== Model Reference → Model ID / Endpoint llm/e2e-distinct-2-simulated → e2e-distinct-2-simulated (Pending) llm/e2e-distinct-simulated → e2e-distinct-simulated (Pending) llm/e2e-embedding-simulated → e2e-embedding-simulated (Pending) llm/e2e-trlp-test-simulated → e2e-trlp-test-simulated (Pending) llm/e2e-unconfigured-facebook-opt-125m-simulated → e2e-unconfigured-facebook-opt-125m-simulated (Pending) llm/echo-simulator → echo-simulator (Pending) llm/facebook-opt-125m-simulated → facebook-opt-125m-simulated (Ready) Endpoint: https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/ llm/premium-simulated-simulated-premium → premium-simulated-simulated-premium (Ready) Endpoint: https://maas.apps.b473a4036217489213f4.prod.konflux-ocp-ci.dev/ ======================================== Gateway / HTTPRoutes ======================================== --- Gateway --- NAME CLASS ADDRESS PROGRAMMED AGE maas-default-gateway openshift-default maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local True 6m29s --- HTTPRoutes (maas-api) --- NAME HOSTNAMES AGE maas-api-route 3m29s ======================================== Authorino ======================================== --- Authorino pods --- --- ======================================== Subscription Selector Endpoint Validation ======================================== Expected URL (from resolved maas-api deployment namespace): https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select (MAAS_API_DEPLOYMENT_NAMESPACE resolved to: odh-ai-gateway-infra) --- Sample AuthPolicy subscription-info configuration --- Inspecting: openshift-ingress/maas-gateway-auth Actual URL in AuthPolicy: https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select ✅ Request body includes requestedModel field Model reference: / Request body preview: { "groups": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.groups : (has(auth.identity.iss) ? (has(auth.identity.groups) && auth.identity.groups.all(g, g.matches('^[A-Za-z0-9:#._/ -]+$')) ? auth.identity.groups : []) : (has(auth.identity.user.groups) && auth.identity.user.groups.all(g, g.matches('^[A-Za-z0-9:#._/ -]+$')) ? auth.identity.user.groups : [])), "username": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.username : (has(auth.identity.preferred_username) ? auth.identity.preferred_username : (has(auth.identity.sub) ? auth.identity.sub : auth.identity.user.username)), "requestedSubscription": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.subscription : ("x-maas-subscription" in request.headers ? request.headers["x-maas-subscription"] : ""), "requestedModel": (size(request.path.split("/").filter(x, x != "")) >= 2 && request.path.split("/").filter(x, x != "")[0] != "v1" && request.path.split("/").filter(x, x != "")[0] != "maas-api" ? request.path.split("/").filter(x, x != "")[0] + "/" + request.path.split("/").filter(x, x != "")[1] : ("x-gateway-model-name" in request.headers ? request.headers["x-gateway-model-name"] : "")) --- Connectivity test (from kuadrant-system, simulates Authorino) --- curl -vsk -m 10 -X POST 'https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select' -H 'Content-Type: application/json' -d '{}' kubectl run failed or timed out ======================================== DNS Resolution Check ======================================== Resolving: maas-api.odh-ai-gateway-infra.svc.cluster.local nslookup failed ======================================== Configuration Summary ======================================== This summary helps compare local vs CI runs: AITenants (all namespaces): 1 MaaSModelRefs (all namespaces): 8 MaaSSubscriptions (models-as-a-service): 2 MaaSAuthPolicies (models-as-a-service): 2 Generated Kuadrant AuthPolicies: 1 Generated Kuadrant TokenRateLimitPolicies: 3 LLMInferenceServices (all namespaces): 8 ExternalModels - inference (all namespaces): 0 ExternalProviders - inference (all namespaces): 0 EnvoyFilters (openshift-ingress): 7 IPP Deployments (openshift-ingress): 0 NetworkPolicies (openshift-ingress): 8 Subscription selector URL: https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select Test user: system:admin