MaaS platform images: MAAS_API_IMAGE=quay.io/opendatahub/maas-api:latest MAAS_CONTROLLER_IMAGE=quay.io/opendatahub/maas-controller:latest AI_GATEWAY_CONTROLLER_IMAGE=quay.io/opendatahub/odh-ai-gateway-controller@sha256:f358282357211d12f2f1e9db09ec906444d74a797745dceb055429a56674ca29 ---------------------------------------- ai-gateway-controller E2E on OpenShift ---------------------------------------- Checking prerequisites... Prerequisites met — logged in as: system:admin DEPLOY_MODE: kustomize MAAS_API_IMAGE: quay.io/opendatahub/maas-api:latest MAAS_CONTROLLER_IMAGE: quay.io/opendatahub/maas-controller:latest AI_GATEWAY_CONTROLLER_IMAGE: quay.io/opendatahub/odh-ai-gateway-controller@sha256:f358282357211d12f2f1e9db09ec906444d74a797745dceb055429a56674ca29 2026-09-13T19:12:41Z deploy_platform start Deploying MaaS platform via ODH operator... Gateway ingress mode for deploy.sh: ocproute (loadbalancer | ocproute) Using custom MaaS API image: quay.io/opendatahub/maas-api:latest Using custom MaaS controller image: quay.io/opendatahub/maas-controller:latest Deployment mode: kustomize Installing cert-manager and LeaderWorkerSet operators... === Installing cert-manager and LeaderWorkerSet operators === 1. Installing cert-manager operator... namespace/cert-manager-operator created operatorgroup.operators.coreos.com/cert-manager-operator created subscription.operators.coreos.com/openshift-cert-manager-operator created Waiting for Subscription cert-manager-operator/openshift-cert-manager-operator... subscription.operators.coreos.com/openshift-cert-manager-operator condition met Waiting for CSV cert-manager-operator.v1.20.0 to succeed... clusterserviceversion.operators.coreos.com/cert-manager-operator.v1.20.0 condition met cert-manager ready. 2. Installing LeaderWorkerSet operator... namespace/openshift-lws-operator created operatorgroup.operators.coreos.com/leader-worker-set created subscription.operators.coreos.com/leader-worker-set created Waiting for Subscription openshift-lws-operator/leader-worker-set... subscription.operators.coreos.com/leader-worker-set condition met Waiting for CSV leader-worker-set.v1.0.0 to succeed... clusterserviceversion.operators.coreos.com/leader-worker-set.v1.0.0 condition met LeaderWorkerSet operator ready. 3. Activating LeaderWorkerSet API... leaderworkersetoperator.operator.openshift.io/cluster created LeaderWorkerSetOperator CR applied. === Done === Verify: oc get pods -n cert-manager-operator oc get pods -n openshift-lws-operator oc get crd leaderworkersets.leaderworkerset.x-k8s.io Installing OpenDataHub operator... === Installing OpenDataHub operator === 1. Setting up ODH catalog... Using community-operators 2. Installing ODH operator... [INFO] Installing operator: opendatahub-operator in namespace: opendatahub [INFO] Creating namespace: opendatahub namespace/opendatahub created namespace/opendatahub condition met [INFO] Creating OperatorGroup in opendatahub for AllNamespaces mode operatorgroup.operators.coreos.com/opendatahub-operatorgroup created [INFO] Creating Subscription for opendatahub-operator from community-operators (channel: fast-3, installPlanApproval: Manual, startingCSV: opendatahub-operator.v3.5.0-ea.2) subscription.operators.coreos.com/opendatahub-operator created [INFO] Manual Subscription: approving initial InstallPlan so first install can proceed... [INFO] Approving initial InstallPlan install-st4tw (Manual subscription) installplan.operators.coreos.com/install-st4tw patched [INFO] Waiting for subscription to install... * Waiting for Subscription opendatahub/opendatahub-operator to start setup... subscription.operators.coreos.com/opendatahub-operator condition met * Waiting for Subscription setup to finish setup. CSV = opendatahub-operator.v3.5.0-ea.2 ... clusterserviceversion.operators.coreos.com/opendatahub-operator.v3.5.0-ea.2 condition met [INFO] Operator opendatahub-operator installed successfully 3. Skipping operator image patch (OPERATOR_IMAGE not set) 4. Waiting for operator CRDs... ⏳ Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 180s)… ✅ CRD datascienceclusters.datasciencecluster.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met 5. Waiting for operator webhook... * Waiting for deployment/opendatahub-operator-controller-manager in opendatahub (timeout: 120s)... * Found deployment/opendatahub-operator-controller-manager deployment.apps/opendatahub-operator-controller-manager condition met 6. Applying DSCInitialization... dscinitialization.dscinitialization.opendatahub.io/default-dsci created Waiting for DSCInitialization to be Ready... Waiting for DSCInitialization Ready (attempt 1/30, phase=Progressing, Ready=unknown)... DSCInitialization is Ready 7. Applying DataScienceCluster... datasciencecluster.datasciencecluster.opendatahub.io/default-dsc serverside-applied 8. Waiting for DataScienceCluster (KServe)... * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... - KServe state: , KserveReady: , ModelsAsServiceReady: (informational only), AIGatewayReady: - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: False (informational only), AIGatewayReady: - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: False (informational only), AIGatewayReady: * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' 9. Waiting for odh-model-controller webhook... ⏳ Waiting for validating webhooks in namespace opendatahub (timeout: 180s)... ✅ Webhook service opendatahub/kserve-webhook-server-service has ready endpoints ✅ Webhook service opendatahub/llmisvc-webhook-server-service has ready endpoints ✅ Webhook service opendatahub/odh-model-controller-webhook-service has ready endpoints ✅ Webhook service opendatahub/opendatahub-operator-controller-manager-service has ready endpoints 🎉 All validating webhook services in opendatahub are ready === ODH installation complete === Verify: kubectl get datasciencecluster -A kubectl get pods -n opendatahub kubectl get pods -n kserve Using policy engine: rhcl (Authorino namespace: kuadrant-system) [INFO] =================================================== [INFO] Models-as-a-Service Deployment [INFO] =================================================== [INFO] Validating configuration... [INFO] Configuration validated successfully [INFO] Deployment configuration: [INFO] Mode: kustomize [INFO] Policy Engine: rhcl [INFO] Namespace: opendatahub [INFO] TLS Backend: true [INFO] External OIDC: false [INFO] MaaS API image: quay.io/opendatahub/maas-api:latest [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:latest [INFO] Starting kustomize-based deployment... [INFO] Installing policy engine: rhcl [INFO] Installing RHCL (Red Hat Connectivity Link - downstream) [INFO] Using RHCL channel head from redhat-operators (stable) [INFO] Installing RHCL into namespace: kuadrant-system [INFO] Installing operator: rhcl-operator in namespace: kuadrant-system [INFO] Creating namespace: kuadrant-system namespace/kuadrant-system created namespace/kuadrant-system condition met [INFO] Creating OperatorGroup in kuadrant-system for AllNamespaces mode operatorgroup.operators.coreos.com/kuadrant-system-operatorgroup created [INFO] Creating Subscription for rhcl-operator from redhat-operators (channel: stable) subscription.operators.coreos.com/rhcl-operator created [INFO] Waiting for subscription to install... * Waiting for Subscription kuadrant-system/rhcl-operator to start setup... subscription.operators.coreos.com/rhcl-operator condition met * Waiting for Subscription setup to finish setup. CSV = rhcl-operator.v1.4.3 ... clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 condition met [INFO] Operator rhcl-operator installed successfully [INFO] Patching rhcl-operator CSV (Gateway API, rate limit failure modes, auth service timeout)... clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched [INFO] CSV patched (Gateway controller and/or rate limit failure modes and/or auth timeout) [INFO] Forcing operator restart to apply CSV env configuration... pod "kuadrant-operator-controller-manager-67c5dcb48c-wkrx9" force deleted pod "kuadrant-operator-controller-manager-76f8f686bf-tct6j" force deleted pod "limitador-operator-controller-manager-74495c69f8-zkhln" force deleted [INFO] Waiting for operator pod to restart... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... deployment "kuadrant-operator-controller-manager" successfully rolled out [WARN] Operator pod may not have correct env yet (ISTIO / RATELIMIT_* failure modes / AUTH_SERVICE_TIMEOUT) [INFO] Waiting 15s for operator to fully initialize with Gateway controller configuration... [INFO] Initializing Gateway API and ModelsAsAService gateway... [INFO] =================================================== [INFO] MaaS Gateway Setup [INFO] =================================================== [INFO] Validating gateway configuration... [INFO] Configuration validated [INFO] Ingress mode: ocproute [INFO] Disconnected: false [INFO] AllowedRoutes: namespaces=opendatahub,odh-ai-gateway-infra,llm [INFO] Detecting cluster domain... [INFO] Detected cluster domain: apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com [INFO] Setting up GatewayClass... [INFO] Creating GatewayClass openshift-default... gatewayclass.gateway.networking.k8s.io/openshift-default created [INFO] Setting up Gateway in ocproute mode (ClusterIP with OpenShift Route)... [INFO] Creating ConfigMap gw-options... configmap/gw-options created [INFO] Creating/updating Gateway maas-default-gateway (ocproute mode)... gateway.gateway.networking.k8s.io/maas-default-gateway serverside-applied [INFO] Waiting for Gateway to be Programmed (timeout: 120s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met [INFO] Gateway is Programmed [INFO] Gateway Service ready: maas-default-gateway-openshift-default [INFO] Creating Route maas-gateway-route... [INFO] Host: maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com [INFO] Target Service: maas-default-gateway-openshift-default route.route.openshift.io/maas-gateway-route created [INFO] Waiting for Route to be Admitted... [INFO] Route is Admitted [INFO] ClusterIP mode setup complete [INFO] [INFO] =================================================== [INFO] Gateway setup completed successfully [INFO] =================================================== [INFO] Mode: ocproute [INFO] Gateway: maas-default-gateway [INFO] Namespace: openshift-ingress [INFO] Route: maas-gateway-route [INFO] Hostname: maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com [INFO] [INFO] Verify with: [INFO] kubectl get gateway maas-default-gateway -n openshift-ingress [INFO] kubectl get route maas-gateway-route -n openshift-ingress [INFO] Applying Kuadrant custom resource in kuadrant-system... kuadrant.kuadrant.io/kuadrant created [INFO] Waiting for Kuadrant to become ready (initial check)... [INFO] Waiting for: Kuadrant ready in kuadrant-system (timeout: 60s) [INFO] Kuadrant ready in kuadrant-system - Ready [INFO] Kuadrant setup complete [INFO] Ensuring namespace exists: opendatahub [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [WARN] DEPLOYING POC POSTGRESQL — NOT INTENDED FOR PRODUCTION USE [WARN] Data is stored in ephemeral storage and will be lost on pod restart. [WARN] For production, use --postgres-connection with an external database [WARN] (AWS RDS, Crunchy Operator, Azure Database, etc.) [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ⚠️ WARNING FOR PRODUCTION USE. ⚠️ ┃ ┃ This deploys PostgreSQL with ephemeral storage (emptyDir). ┃ ┃ Data WILL be lost on pod restart. ┃ ┃ For production, use an external database: ┃ ┃ deploy.sh --postgres-connection postgresql://... ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 🔧 Fresh install: Deploying PostgreSQL in infrastructure namespace 'odh-ai-gateway-infra'... 📦 Creating infrastructure namespace 'odh-ai-gateway-infra'... namespace/odh-ai-gateway-infra created Generated random PostgreSQL password (stored in secret postgres-creds) Creating PostgreSQL deployment... ⚠️ Using POC configuration (ephemeral storage) Using default PostgreSQL image (operator CSV not available) Image: registry.redhat.io/rhel9/postgresql-16:latest secret/postgres-creds created deployment.apps/postgres created service/postgres created secret/maas-db-config created Waiting for PostgreSQL to be ready... deployment.apps/postgres condition met ✅ PostgreSQL deployed successfully Namespace: odh-ai-gateway-infra Database: maas User: maas Secret: maas-db-config (contains DB_CONNECTION_URL with FQDN) ⚠️ For production, use AWS RDS, Crunchy Operator, or Azure Database Note: Schema migrations run automatically when maas-api starts [INFO] Configuring TLS backend for Authorino and MaaS API... * Waiting for deployment/authorino in kuadrant-system (timeout: 300s)... * Found deployment/authorino [INFO] Running TLS configuration script... [INFO] TLS configuration script completed successfully [INFO] Restarting deployments to pick up TLS configuration... deployment.apps/authorino restarted [INFO] Waiting for Authorino deployment to be ready... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out [INFO] TLS backend configuration complete [INFO] Kustomize prerequisite deployment completed [INFO] [INFO] MaaS Controller... [INFO] maas-controller source tree not present; using synced deployment/ manifests (kustomize mode) [INFO] Phase 1: Applying MaaS CRDs and waiting until Established (controller creates Config after CRD is ready)... ⏳ Applying MaaS Controller CRDs from /workspace/source/scripts/../deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io created ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] Phase 2: Applying full controller kustomize (same as operator: deployment/base/maas-controller/default)... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io configured serviceaccount/maas-controller created role.rbac.authorization.k8s.io/maas-controller-leader-election-role created clusterrole.rbac.authorization.k8s.io/maas-controller-cluster-config-role created clusterrole.rbac.authorization.k8s.io/maas-controller-ocp-role created clusterrole.rbac.authorization.k8s.io/maas-controller-role created clusterrole.rbac.authorization.k8s.io/maas-owner-role created clusterrole.rbac.authorization.k8s.io/maas-viewer-role created rolebinding.rbac.authorization.k8s.io/maas-controller-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-cluster-config-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-ocp-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-rolebinding created configmap/maas-parameters created service/maas-controller-metrics created service/maas-controller-webhook-service created deployment.apps/maas-controller created servicemonitor.monitoring.coreos.com/maas-controller-metrics created networkpolicy.networking.k8s.io/maas-controller-allow-monitoring created validatingwebhookconfiguration.admissionregistration.k8s.io/maas-validating-webhook-configuration created [INFO] Restarting maas-controller to pick up manifest and ConfigMap changes deployment.apps/maas-controller restarted [INFO] Waiting for maas-controller to be ready... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... deployment "maas-controller" successfully rolled out [INFO] Controller ready. [INFO] [INFO] Waiting for Tenant reconciler to deploy maas-api... [INFO] Applying secret migration RBAC to namespace odh-ai-gateway-infra... role.rbac.authorization.k8s.io/maas-controller-secret-migrate created rolebinding.rbac.authorization.k8s.io/maas-controller-secret-migrate created [INFO] maas-api deployment found in odh-ai-gateway-infra, waiting for rollout... Waiting for deployment "maas-api" rollout to finish: 0 of 1 updated replicas are available... deployment "maas-api" successfully rolled out [INFO] maas-api is ready [INFO] [INFO] MaaS API and MaaS Controller deployment completed successfully! [INFO] maas-api image: quay.io/opendatahub/maas-api:latest (namespace: odh-ai-gateway-infra) [INFO] maas-controller image: quay.io/opendatahub/maas-controller:latest (namespace: opendatahub) [INFO] =================================================== [INFO] Models-as-a-Service Deployment completed successfully! [INFO] =================================================== * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' ⚠️ WARNING: Skipping Authorino readiness check (SKIP_AUTH_CHECK=true) ✅ MaaS platform deployment completed 2026-09-13T19:18:19Z deploy_platform end ---------------------------------------- Deploying Models ---------------------------------------- 2026-09-13T19:18:19Z deploy_models start Deploying MaaS system (free + premium: LLMIS + MaaSModelRef + MaaSAuthPolicy + MaaSSubscription) Waiting for Gateway openshift-ingress/maas-default-gateway to be Programmed=True (timeout: 600s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met ✅ Gateway openshift-ingress/maas-default-gateway is Programmed Creating 'llm' namespace... namespace/llm created 'models-as-a-service' namespace already exists maasauthpolicy.maas.opendatahub.io/premium-simulator-access created maasauthpolicy.maas.opendatahub.io/simulator-access created maasmodelref.maas.opendatahub.io/e2e-distinct-2-simulated created maasmodelref.maas.opendatahub.io/e2e-distinct-simulated created maasmodelref.maas.opendatahub.io/e2e-embedding-simulated created maasmodelref.maas.opendatahub.io/e2e-trlp-test-simulated created maasmodelref.maas.opendatahub.io/e2e-unconfigured-facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/premium-simulated-simulated-premium created maassubscription.maas.opendatahub.io/premium-simulator-subscription created maassubscription.maas.opendatahub.io/simulator-subscription created llminferenceservice.serving.kserve.io/e2e-distinct-2-simulated created llminferenceservice.serving.kserve.io/e2e-distinct-simulated created llminferenceservice.serving.kserve.io/e2e-embedding-simulated created llminferenceservice.serving.kserve.io/e2e-trlp-test-simulated created llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium created ✅ MaaS system deployed (free + premium + e2e test fixtures) Waiting for models to be ready (timeout: 300s)... llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated condition met llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium condition met llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated condition met ✅ Simulator models ready Waiting for governed MaaSModelRefs to be Ready (timeout: 300s)... ✅ Governed MaaSModelRefs ready Waiting for MaaS Kuadrant AuthPolicies to be enforced (selector: app.kubernetes.io/managed-by=maas-controller, timeout: 180s)... ✅ All MaaS AuthPolicies enforced (1 policies) 2026-09-13T19:21:23Z deploy_models end Patching Authorino to log_level DEBUG... authorino.operator.authorino.kuadrant.io/authorino patched ✅ Authorino patched to log_level DEBUG Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out ---------------------------------------- Deploying ai-gateway-controller ---------------------------------------- 2026-09-13T19:21:25Z deploy_ai_gateway_controller start Deploying ai-gateway-controller image: quay.io/opendatahub/odh-ai-gateway-controller@sha256:f358282357211d12f2f1e9db09ec906444d74a797745dceb055429a56674ca29 praxis-extproc image: quay.io/opendatahub/odh-praxis-extproc:odh-stable namespace: opendatahub gateway: openshift-ingress/maas-default-gateway remove maas IPP: true Pausing maas-controller (scale 1 -> 0) to avoid IPP reconcile during handoff ... deployment.apps/maas-controller scaled deployment "maas-controller" successfully rolled out Removing maas-controller legacy IPP in openshift-ingress ... deployment.apps "payload-processing" deleted service "payload-processing" deleted destinationrule.networking.istio.io "payload-processing" deleted deployment.apps "payload-pre-processing" deleted service "payload-pre-processing" deleted destinationrule.networking.istio.io "payload-pre-processing" deleted envoyfilter.networking.istio.io "payload-processing" deleted networkpolicy.networking.k8s.io "payload-processing" deleted No resources found Legacy IPP Deployments removed from openshift-ingress serviceaccount/ai-gateway-controller created clusterrole.rbac.authorization.k8s.io/ai-gateway-controller-role created clusterrolebinding.rbac.authorization.k8s.io/ai-gateway-controller-rolebinding created configmap/ai-gateway-controller-parameters created deployment.apps/ai-gateway-controller created Waiting for deployment "ai-gateway-controller" rollout to finish: 0 of 1 updated replicas are available... deployment "ai-gateway-controller" successfully rolled out Triggering immediate praxis-extproc install ... deployment.apps/ai-gateway-controller restarted Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... deployment "ai-gateway-controller" successfully rolled out Waiting for praxis-extproc (quay.io/opendatahub/odh-praxis-extproc:odh-stable) in openshift-ingress (timeout: 180s) ... ✅ praxis-extproc ready: openshift-ingress/payload-processing image=quay.io/opendatahub/odh-praxis-extproc:odh-stable Annotating praxis IPP resources opendatahub.io/managed=false so maas-controller skips them ... deployment.apps/payload-processing annotated service/payload-processing annotated destinationrule.networking.istio.io/payload-processing annotated deployment.apps/payload-pre-processing annotated service/payload-pre-processing annotated destinationrule.networking.istio.io/payload-pre-processing annotated envoyfilter.networking.istio.io/payload-processing annotated networkpolicy.networking.k8s.io/payload-processing annotated Resuming maas-controller (scale -> 1) ... deployment.apps/maas-controller scaled Waiting for deployment "maas-controller" rollout to finish: 0 of 1 updated replicas are available... deployment "maas-controller" successfully rolled out Verified ext_proc dataplane image: quay.io/opendatahub/odh-praxis-extproc:odh-stable ai-gateway-controller rollout complete (praxis-extproc handoff done) 2026-09-13T19:22:07Z deploy_ai_gateway_controller end ---------------------------------------- Setting up variables for tests ---------------------------------------- -- Setting up variables for tests -- HOST: maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com MAAS_API_BASE_URL: https://maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com/maas-api ---------------------------------------- Setting up test tokens ---------------------------------------- Setting up premium test token (SA-based, works when oc whoami -t is unavailable)... Creating namespace: premium-users-namespace namespace/premium-users-namespace created Creating service account: premium-service-account serviceaccount/premium-service-account created Patching MaaSAuthPolicy premium-simulator-access to include system:serviceaccount:premium-users-namespace:premium-service-account... maasauthpolicy.maas.opendatahub.io/premium-simulator-access patched Patching MaaSSubscription premium-simulator-subscription to include system:serviceaccount:premium-users-namespace:premium-service-account... maassubscription.maas.opendatahub.io/premium-simulator-subscription patched Waiting for MaaSSubscriptions to reconcile after patch (timeout: 60s)... ✅ Both subscriptions ready: simulator-subscription=Active, premium-simulator-subscription=Active ✅ Premium test token setup complete (E2E_TEST_TOKEN_SA_* exported) Setting up test tokens (admin + regular user)... Current admin session: system:admin (will be preserved) ⚠️ No htpasswd token available - using SA token (admin tests may fail) Creating namespace: maas-admin namespace/maas-admin created Creating service account: tester-admin-user in maas-admin serviceaccount/tester-admin-user created Creating cluster role binding for tester-admin-user clusterrolebinding.rbac.authorization.k8s.io/tester-admin-user-binding created ✅ User setup completed: tester-admin-user (namespace: maas-admin) role.rbac.authorization.k8s.io/maas-admin-e2e created rolebinding.rbac.authorization.k8s.io/maas-admin-e2e-system-serviceaccount-maas-admin-tester-admin-user created auth.services.platform.opendatahub.io/auth patched ✅ Added system:serviceaccounts:maas-admin to Auth CR adminGroups (SA admin fallback) clusterrole.rbac.authorization.k8s.io/maas-admin created rolebinding.rbac.authorization.k8s.io/odh-admins-maas-admin created Creating separate SA token for regular user (required for IDOR tests)... Creating service account: tester-regular-user in default serviceaccount/tester-regular-user created Creating cluster role binding for tester-regular-user clusterrolebinding.rbac.authorization.k8s.io/tester-regular-user-binding created ✅ User setup completed: tester-regular-user (namespace: default) ✅ Regular user token for tester-regular-user (SA-based, namespace: default) Token setup complete (main session unchanged: system:admin) ---------------------------------------- Validating Deployment ---------------------------------------- 2026-09-13T19:22:11Z validate start Deployment Validation ========================================= 🚀 MaaS Platform Deployment Validation ========================================= ========================================= 1️⃣ Component Status Checks ========================================= 🔍 Checking: MaaS API pods ✅ PASS: MaaS API has 1 running pod(s) 🔍 Checking: Policy engine pods (RHCL/Kuadrant) ✅ PASS: Policy engine has 7 running pod(s) in kuadrant-system 🔍 Checking: OpenDataHub/KServe pods ℹ️ opendatahub namespace: 9 running pod(s) ✅ PASS: OpenDataHub/RHOAI has 9 total running pod(s) 🔍 Checking: LLM namespace and models ✅ PASS: Found 7 LLMInferenceService(s) with 7 running pod(s) ========================================= 2️⃣ Gateway Status ========================================= 🔍 Checking: Gateway resource ✅ PASS: Gateway is Accepted and Programmed 🔍 Checking: HTTPRoute for maas-api ✅ PASS: HTTPRoute maas-api-route is configured and accepted 🔍 Checking: Gateway hostname ✅ PASS: Gateway hostname: https://maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com ========================================= 3️⃣ Policy Status ========================================= 🔍 Checking: AuthPolicy ✅ PASS: AuthPolicy is configured and accepted 🔍 Checking: TokenRateLimitPolicy ✅ PASS: TokenRateLimitPolicy is configured and accepted ========================================= 4️⃣ API Endpoint Tests ========================================= ℹ️ Using gateway endpoint: https://maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com 🔍 Checking: Authentication token ✅ PASS: OpenShift identity token available 🔍 Checking: MaaS API key creation ✅ PASS: MaaS API key created (name: validate-test-1789327333) 🔍 Checking: Models endpoint ℹ️ Testing: curl -sSk https://maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com/maas-api/v1/models -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" ✅ PASS: Models endpoint accessible, found 1 model(s) ℹ️ Available models: • publishers/llm/models/facebook/opt-125m - https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ ℹ️ Using first available model: publishers/llm/models/facebook/opt-125m for validation ℹ️ Rewrote internal model URL to external gateway: https://maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com/ 🔍 Checking: Model inference endpoint ℹ️ Testing: curl -sSk -X POST https://maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com/v1/chat/completions -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"model": "publishers/llm/models/facebook/opt-125m", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50}' ❌ FAIL: Model inference endpoint not found (HTTP 404)  Reason: Path is incorrect - traffic reaching but wrong path  Suggestion: Check model HTTPRoute configuration: kubectl get httproute -n llm && kubectl describe llminferenceservice -n llm 🔍 Checking: Rate limiting ℹ️ User tier: unknown (could not extract from token) ℹ️ Sending 10 rapid requests to test rate limiting... ℹ️ Request 1 failed with HTTP 404 ℹ️ Request 2 failed with HTTP 404 ℹ️ Request 3 failed with HTTP 404 ❌ FAIL: Rate limiting test failed  Reason: All 10 requests failed (got 10 errors)  Suggestion: Check TokenRateLimitPolicy, Limitador, and auth service health 🔍 Checking: Authorization enforcement (401 without token) ✅ PASS: Authorization is enforced (got 401 without token) ========================================= 📊 Validation Summary ========================================= Results: ✅ Passed: 13 ❌ Failed: 2 ⚠️ Warnings: 0 ❌ FAIL: Some checks failed. Please review the errors above. Common fixes: - Wait for pods to start: kubectl get pods -A | grep -v Running - Check operator logs: kubectl logs -n kuadrant-system -l app.kubernetes.io/name=kuadrant-operator - Re-run deployment: ./scripts/deploy.sh Usage: ./scripts/validate-deployment.sh [MODEL_NAME] MODEL_NAME: Optional. Specify a model to validate against First validation failed; retrying after short wait... Waiting for Gateway openshift-ingress/maas-default-gateway to be Programmed=True (timeout: 60s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met ✅ Gateway openshift-ingress/maas-default-gateway is Programmed deployment.apps/maas-controller condition met deployment.apps/maas-api condition met ========================================= 🚀 MaaS Platform Deployment Validation ========================================= ========================================= 1️⃣ Component Status Checks ========================================= 🔍 Checking: MaaS API pods ✅ PASS: MaaS API has 1 running pod(s) 🔍 Checking: Policy engine pods (RHCL/Kuadrant) ✅ PASS: Policy engine has 7 running pod(s) in kuadrant-system 🔍 Checking: OpenDataHub/KServe pods ℹ️ opendatahub namespace: 9 running pod(s) ✅ PASS: OpenDataHub/RHOAI has 9 total running pod(s) 🔍 Checking: LLM namespace and models ✅ PASS: Found 7 LLMInferenceService(s) with 7 running pod(s) ========================================= 2️⃣ Gateway Status ========================================= 🔍 Checking: Gateway resource ✅ PASS: Gateway is Accepted and Programmed 🔍 Checking: HTTPRoute for maas-api ✅ PASS: HTTPRoute maas-api-route is configured and accepted 🔍 Checking: Gateway hostname ✅ PASS: Gateway hostname: https://maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com ========================================= 3️⃣ Policy Status ========================================= 🔍 Checking: AuthPolicy ✅ PASS: AuthPolicy is configured and accepted 🔍 Checking: TokenRateLimitPolicy ✅ PASS: TokenRateLimitPolicy is configured and accepted ========================================= 4️⃣ API Endpoint Tests ========================================= ℹ️ Using gateway endpoint: https://maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com 🔍 Checking: Authentication token ✅ PASS: OpenShift identity token available 🔍 Checking: MaaS API key creation ✅ PASS: MaaS API key created (name: validate-test-1789327338) 🔍 Checking: Models endpoint ℹ️ Testing: curl -sSk https://maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com/maas-api/v1/models -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" ✅ PASS: Models endpoint accessible, found 1 model(s) ℹ️ Available models: • publishers/llm/models/facebook/opt-125m - https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ ℹ️ Using first available model: publishers/llm/models/facebook/opt-125m for validation ℹ️ Rewrote internal model URL to external gateway: https://maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com/ 🔍 Checking: Model inference endpoint ℹ️ Testing: curl -sSk -X POST https://maas.apps.4b1b33d7-03a3-4926-8e17-83c148810f54.prod.konfluxeaas.com/v1/chat/completions -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"model": "publishers/llm/models/facebook/opt-125m", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50}' ❌ FAIL: Model inference endpoint not found (HTTP 404)  Reason: Path is incorrect - traffic reaching but wrong path  Suggestion: Check model HTTPRoute configuration: kubectl get httproute -n llm && kubectl describe llminferenceservice -n llm 🔍 Checking: Rate limiting ℹ️ User tier: unknown (could not extract from token) ℹ️ Sending 10 rapid requests to test rate limiting... ℹ️ Request 1 failed with HTTP 404 ℹ️ Request 2 failed with HTTP 404 ℹ️ Request 3 failed with HTTP 404 ❌ FAIL: Rate limiting test failed  Reason: All 10 requests failed (got 10 errors)  Suggestion: Check TokenRateLimitPolicy, Limitador, and auth service health 🔍 Checking: Authorization enforcement (401 without token) ✅ PASS: Authorization is enforced (got 401 without token) ========================================= 📊 Validation Summary ========================================= Results: ✅ Passed: 13 ❌ Failed: 2 ⚠️ Warnings: 0 ❌ FAIL: Some checks failed. Please review the errors above. Common fixes: - Wait for pods to start: kubectl get pods -A | grep -v Running - Check operator logs: kubectl logs -n kuadrant-system -l app.kubernetes.io/name=kuadrant-operator - Re-run deployment: ./scripts/deploy.sh Usage: ./scripts/validate-deployment.sh [MODEL_NAME] MODEL_NAME: Optional. Specify a model to validate against ERROR: Deployment validation failed after retry Deploy failed; attempting to resume maas-controller ... Resuming maas-controller (scale -> 1) ... deployment.apps/maas-controller scaled deployment "maas-controller" successfully rolled out