Fetching https://github.com/ryancham715/models-as-a-service@5c36d49a5c1bcd668e9f5bb995a7599bceb6125a into /workspace/source/test/maas-e2e ... MaaS checkout at: 5c36d49a5c1bcd668e9f5bb995a7599bceb6125a Patched /workspace/source/test/maas-e2e/scripts/deploy.sh Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS checkout fixture root) Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS AuthPolicy wait scope) MaaS platform images: MAAS_API_IMAGE=quay.io/opendatahub/maas-api:latest MAAS_CONTROLLER_IMAGE=quay.io/opendatahub/maas-controller:latest AI_GATEWAY_CONTROLLER_IMAGE=quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-37 PRAXIS_EXTPROC_IMAGE=quay.io/opendatahub/odh-praxis-extproc:odh-stable ---------------------------------------- ai-gateway-controller E2E on OpenShift ---------------------------------------- Checking prerequisites... Prerequisites met — logged in as: system:admin DEPLOY_MODE: kustomize MAAS_API_IMAGE: quay.io/opendatahub/maas-api:latest MAAS_CONTROLLER_IMAGE: quay.io/opendatahub/maas-controller:latest AI_GATEWAY_CONTROLLER_IMAGE: quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-37 PRAXIS_EXTPROC_IMAGE: quay.io/opendatahub/odh-praxis-extproc:odh-stable 2026-09-19T01:44:38Z deploy_platform start Deploying MaaS platform via ODH operator... Gateway ingress mode for deploy.sh: ocproute (loadbalancer | ocproute) Using custom MaaS API image: quay.io/opendatahub/maas-api:latest Using custom MaaS controller image: quay.io/opendatahub/maas-controller:latest Deployment mode: kustomize Installing cert-manager and LeaderWorkerSet operators... === Installing cert-manager and LeaderWorkerSet operators === 1. Installing cert-manager operator... namespace/cert-manager-operator created operatorgroup.operators.coreos.com/cert-manager-operator created subscription.operators.coreos.com/openshift-cert-manager-operator created Waiting for Subscription cert-manager-operator/openshift-cert-manager-operator... subscription.operators.coreos.com/openshift-cert-manager-operator condition met Waiting for CSV cert-manager-operator.v1.20.0 to succeed... clusterserviceversion.operators.coreos.com/cert-manager-operator.v1.20.0 condition met cert-manager ready. 2. Installing LeaderWorkerSet operator... namespace/openshift-lws-operator created operatorgroup.operators.coreos.com/leader-worker-set created subscription.operators.coreos.com/leader-worker-set created Waiting for Subscription openshift-lws-operator/leader-worker-set... subscription.operators.coreos.com/leader-worker-set condition met Waiting for CSV leader-worker-set.v1.0.0 to succeed... clusterserviceversion.operators.coreos.com/leader-worker-set.v1.0.0 condition met LeaderWorkerSet operator ready. 3. Activating LeaderWorkerSet API... leaderworkersetoperator.operator.openshift.io/cluster created LeaderWorkerSetOperator CR applied. === Done === Verify: oc get pods -n cert-manager-operator oc get pods -n openshift-lws-operator oc get crd leaderworkersets.leaderworkerset.x-k8s.io Installing OpenDataHub operator... === Installing OpenDataHub operator === 1. Setting up ODH catalog... Using community-operators 2. Installing ODH operator... [INFO] Installing operator: opendatahub-operator in namespace: opendatahub [INFO] Creating namespace: opendatahub namespace/opendatahub created namespace/opendatahub condition met [INFO] Creating OperatorGroup in opendatahub for AllNamespaces mode operatorgroup.operators.coreos.com/opendatahub-operatorgroup created [INFO] Creating Subscription for opendatahub-operator from community-operators (channel: fast-3, installPlanApproval: Manual, startingCSV: opendatahub-operator.v3.5.0-ea.2) subscription.operators.coreos.com/opendatahub-operator created [INFO] Manual Subscription: approving initial InstallPlan so first install can proceed... [INFO] Approving initial InstallPlan install-xz5cm (Manual subscription) installplan.operators.coreos.com/install-xz5cm patched [INFO] Waiting for subscription to install... * Waiting for Subscription opendatahub/opendatahub-operator to start setup... subscription.operators.coreos.com/opendatahub-operator condition met * Waiting for Subscription setup to finish setup. CSV = opendatahub-operator.v3.5.0-ea.2 ... clusterserviceversion.operators.coreos.com/opendatahub-operator.v3.5.0-ea.2 condition met [INFO] Operator opendatahub-operator installed successfully 3. Skipping operator image patch (OPERATOR_IMAGE not set) 4. Waiting for operator CRDs... ⏳ Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 180s)… ✅ CRD datascienceclusters.datasciencecluster.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met 5. Waiting for operator webhook... * Waiting for deployment/opendatahub-operator-controller-manager in opendatahub (timeout: 120s)... * Found deployment/opendatahub-operator-controller-manager deployment.apps/opendatahub-operator-controller-manager condition met 6. Applying DSCInitialization... dscinitialization.dscinitialization.opendatahub.io/default-dsci created Waiting for DSCInitialization to be Ready... Waiting for DSCInitialization Ready (attempt 1/30, phase=Progressing, Ready=unknown)... DSCInitialization is Ready 7. Applying DataScienceCluster... datasciencecluster.datasciencecluster.opendatahub.io/default-dsc serverside-applied 8. Waiting for DataScienceCluster (KServe)... * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... - KServe state: , KserveReady: , ModelsAsServiceReady: (informational only), AIGatewayReady: - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: False (informational only), AIGatewayReady: - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: False (informational only), AIGatewayReady: * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' 9. Waiting for odh-model-controller webhook... ⏳ Waiting for validating webhooks in namespace opendatahub (timeout: 180s)... ✅ Webhook service opendatahub/kserve-webhook-server-service has ready endpoints ✅ Webhook service opendatahub/llmisvc-webhook-server-service has ready endpoints ✅ Webhook service opendatahub/odh-model-controller-webhook-service has ready endpoints ✅ Webhook service opendatahub/opendatahub-operator-controller-manager-service has ready endpoints 🎉 All validating webhook services in opendatahub are ready === ODH installation complete === Verify: kubectl get datasciencecluster -A kubectl get pods -n opendatahub kubectl get pods -n kserve Using policy engine: rhcl (Authorino namespace: kuadrant-system) [INFO] =================================================== [INFO] Models-as-a-Service Deployment [INFO] =================================================== [INFO] Validating configuration... [INFO] Configuration validated successfully [INFO] Deployment configuration: [INFO] Mode: kustomize [INFO] Policy Engine: rhcl [INFO] Namespace: opendatahub [INFO] TLS Backend: true [INFO] External OIDC: false [INFO] MaaS API image: quay.io/opendatahub/maas-api:latest [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:latest [INFO] Starting kustomize-based deployment... [INFO] Installing policy engine: rhcl [INFO] Installing RHCL (Red Hat Connectivity Link - downstream) [INFO] Using RHCL channel head from redhat-operators (stable) [INFO] Installing RHCL into namespace: kuadrant-system [INFO] Installing operator: rhcl-operator in namespace: kuadrant-system [INFO] Creating namespace: kuadrant-system namespace/kuadrant-system created namespace/kuadrant-system condition met [INFO] Creating OperatorGroup in kuadrant-system for AllNamespaces mode operatorgroup.operators.coreos.com/kuadrant-system-operatorgroup created [INFO] Creating Subscription for rhcl-operator from redhat-operators (channel: stable) subscription.operators.coreos.com/rhcl-operator created [INFO] Waiting for subscription to install... * Waiting for Subscription kuadrant-system/rhcl-operator to start setup... subscription.operators.coreos.com/rhcl-operator condition met * Waiting for Subscription setup to finish setup. CSV = rhcl-operator.v1.4.3 ... clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 condition met [INFO] Operator rhcl-operator installed successfully [INFO] Patching rhcl-operator CSV (Gateway API, rate limit failure modes, auth service timeout)... clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched [INFO] CSV patched (Gateway controller and/or rate limit failure modes and/or auth timeout) [INFO] Forcing operator restart to apply CSV env configuration... pod "kuadrant-operator-controller-manager-76f8f686bf-8nz5h" force deleted pod "kuadrant-operator-controller-manager-7978cbdfbc-ql6lb" force deleted pod "limitador-operator-controller-manager-74495c69f8-xshxd" force deleted [INFO] Waiting for operator pod to restart... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... deployment "kuadrant-operator-controller-manager" successfully rolled out [INFO] Operator pod has required CSV env (ISTIO gateway controller + RATELIMIT_* failure modes + AUTH_SERVICE_TIMEOUT) [INFO] Waiting 15s for operator to fully initialize with Gateway controller configuration... [INFO] Initializing Gateway API and ModelsAsAService gateway... [INFO] =================================================== [INFO] MaaS Gateway Setup [INFO] =================================================== [INFO] Validating gateway configuration... [INFO] Configuration validated [INFO] Ingress mode: ocproute [INFO] Disconnected: false [INFO] AllowedRoutes: namespaces=opendatahub,odh-ai-gateway-infra,llm [INFO] Detecting cluster domain... [INFO] Detected cluster domain: apps.9bb8ac5d-ce86-4894-b95c-de39a7ac43d6.prod.konfluxeaas.com [INFO] Setting up GatewayClass... [INFO] Creating GatewayClass openshift-default... gatewayclass.gateway.networking.k8s.io/openshift-default created [INFO] Setting up Gateway in ocproute mode (ClusterIP with OpenShift Route)... [INFO] Creating ConfigMap gw-options... configmap/gw-options created [INFO] Creating/updating Gateway maas-default-gateway (ocproute mode)... gateway.gateway.networking.k8s.io/maas-default-gateway serverside-applied [INFO] Waiting for Gateway to be Programmed (timeout: 120s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met [INFO] Gateway is Programmed [INFO] Gateway Service ready: maas-default-gateway-openshift-default [INFO] Creating Route maas-gateway-route... [INFO] Host: maas.apps.9bb8ac5d-ce86-4894-b95c-de39a7ac43d6.prod.konfluxeaas.com [INFO] Target Service: maas-default-gateway-openshift-default route.route.openshift.io/maas-gateway-route created [INFO] Waiting for Route to be Admitted... [INFO] Route is Admitted [INFO] ClusterIP mode setup complete [INFO] [INFO] =================================================== [INFO] Gateway setup completed successfully [INFO] =================================================== [INFO] Mode: ocproute [INFO] Gateway: maas-default-gateway [INFO] Namespace: openshift-ingress [INFO] Route: maas-gateway-route [INFO] Hostname: maas.apps.9bb8ac5d-ce86-4894-b95c-de39a7ac43d6.prod.konfluxeaas.com [INFO] [INFO] Verify with: [INFO] kubectl get gateway maas-default-gateway -n openshift-ingress [INFO] kubectl get route maas-gateway-route -n openshift-ingress [INFO] Applying Kuadrant custom resource in kuadrant-system... kuadrant.kuadrant.io/kuadrant created [INFO] Waiting for Kuadrant to become ready (initial check)... [INFO] Waiting for: Kuadrant ready in kuadrant-system (timeout: 60s) [INFO] Kuadrant ready in kuadrant-system - Ready [INFO] Kuadrant setup complete [INFO] Ensuring namespace exists: opendatahub [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [WARN] DEPLOYING POC POSTGRESQL — NOT INTENDED FOR PRODUCTION USE [WARN] Data is stored in ephemeral storage and will be lost on pod restart. [WARN] For production, use --postgres-connection with an external database [WARN] (AWS RDS, Crunchy Operator, Azure Database, etc.) [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ⚠️ WARNING FOR PRODUCTION USE. ⚠️ ┃ ┃ This deploys PostgreSQL with ephemeral storage (emptyDir). ┃ ┃ Data WILL be lost on pod restart. ┃ ┃ For production, use an external database: ┃ ┃ deploy.sh --postgres-connection postgresql://... ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 🔧 Fresh install: Deploying PostgreSQL in infrastructure namespace 'odh-ai-gateway-infra'... 📦 Creating infrastructure namespace 'odh-ai-gateway-infra'... namespace/odh-ai-gateway-infra created Generated random PostgreSQL password (stored in secret postgres-creds) Creating PostgreSQL deployment... ⚠️ Using POC configuration (ephemeral storage) Using default PostgreSQL image (operator CSV not available) Image: registry.redhat.io/rhel9/postgresql-16:latest secret/postgres-creds created deployment.apps/postgres created service/postgres created secret/maas-db-config created Waiting for PostgreSQL to be ready... deployment.apps/postgres condition met ✅ PostgreSQL deployed successfully Namespace: odh-ai-gateway-infra Database: maas User: maas Secret: maas-db-config (contains DB_CONNECTION_URL with FQDN) ⚠️ For production, use AWS RDS, Crunchy Operator, or Azure Database Note: Schema migrations run automatically when maas-api starts [INFO] Configuring TLS backend for Authorino and MaaS API... * Waiting for deployment/authorino in kuadrant-system (timeout: 300s)... * Found deployment/authorino [INFO] Running TLS configuration script... [INFO] TLS configuration script completed successfully [INFO] Restarting deployments to pick up TLS configuration... deployment.apps/authorino restarted [INFO] Waiting for Authorino deployment to be ready... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out [INFO] TLS backend configuration complete [INFO] Kustomize prerequisite deployment completed [INFO] [INFO] MaaS Controller... [INFO] Phase 1: Applying MaaS CRDs and waiting until Established (controller creates Config after CRD is ready)... ⏳ Applying MaaS Controller CRDs from /workspace/source/test/maas-e2e/scripts/../deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io created ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] Phase 2: Applying full controller kustomize (same as operator: deployment/base/maas-controller/default)... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io configured serviceaccount/maas-controller created role.rbac.authorization.k8s.io/maas-controller-leader-election-role created clusterrole.rbac.authorization.k8s.io/maas-controller-cluster-config-role created clusterrole.rbac.authorization.k8s.io/maas-controller-ocp-role created clusterrole.rbac.authorization.k8s.io/maas-controller-role created clusterrole.rbac.authorization.k8s.io/maas-owner-role created clusterrole.rbac.authorization.k8s.io/maas-viewer-role created rolebinding.rbac.authorization.k8s.io/maas-controller-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-cluster-config-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-ocp-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-rolebinding created configmap/maas-parameters created service/maas-controller-metrics created service/maas-controller-webhook-service created deployment.apps/maas-controller created servicemonitor.monitoring.coreos.com/maas-controller-metrics created networkpolicy.networking.k8s.io/maas-controller-allow-monitoring created validatingwebhookconfiguration.admissionregistration.k8s.io/maas-validating-webhook-configuration created [INFO] Restarting maas-controller to pick up manifest and ConfigMap changes deployment.apps/maas-controller restarted [INFO] Waiting for maas-controller to be ready... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... deployment "maas-controller" successfully rolled out [INFO] Controller ready. [INFO] [INFO] Waiting for Tenant reconciler to deploy maas-api... [INFO] Applying secret migration RBAC to namespace odh-ai-gateway-infra... role.rbac.authorization.k8s.io/maas-controller-secret-migrate created rolebinding.rbac.authorization.k8s.io/maas-controller-secret-migrate created [INFO] maas-api deployment found in odh-ai-gateway-infra, waiting for rollout... Waiting for deployment "maas-api" rollout to finish: 0 of 1 updated replicas are available... deployment "maas-api" successfully rolled out [INFO] maas-api is ready [INFO] [INFO] MaaS API and MaaS Controller deployment completed successfully! [INFO] maas-api image: quay.io/opendatahub/maas-api:latest (namespace: odh-ai-gateway-infra) [INFO] maas-controller image: quay.io/opendatahub/maas-controller:latest (namespace: opendatahub) [INFO] =================================================== [INFO] Models-as-a-Service Deployment completed successfully! [INFO] =================================================== * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' ⚠️ WARNING: Skipping Authorino readiness check (SKIP_AUTH_CHECK=true) ✅ MaaS platform deployment completed 2026-09-19T01:50:02Z deploy_platform end ---------------------------------------- Deploying Models ---------------------------------------- 2026-09-19T01:50:02Z deploy_models start Deploying MaaS system (free + premium: LLMIS + MaaSModelRef + MaaSAuthPolicy + MaaSSubscription) Waiting for Gateway openshift-ingress/maas-default-gateway to be Programmed=True (timeout: 600s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met ✅ Gateway openshift-ingress/maas-default-gateway is Programmed Creating 'llm' namespace... namespace/llm created 'models-as-a-service' namespace already exists maasauthpolicy.maas.opendatahub.io/premium-simulator-access created maasauthpolicy.maas.opendatahub.io/simulator-access created maasmodelref.maas.opendatahub.io/e2e-distinct-2-simulated created maasmodelref.maas.opendatahub.io/e2e-distinct-simulated created maasmodelref.maas.opendatahub.io/e2e-embedding-simulated created maasmodelref.maas.opendatahub.io/e2e-trlp-test-simulated created maasmodelref.maas.opendatahub.io/e2e-unconfigured-facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/premium-simulated-simulated-premium created maassubscription.maas.opendatahub.io/premium-simulator-subscription created maassubscription.maas.opendatahub.io/simulator-subscription created llminferenceservice.serving.kserve.io/e2e-distinct-2-simulated created llminferenceservice.serving.kserve.io/e2e-distinct-simulated created llminferenceservice.serving.kserve.io/e2e-embedding-simulated created llminferenceservice.serving.kserve.io/e2e-trlp-test-simulated created llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium created ✅ MaaS system deployed (free + premium + e2e test fixtures) Waiting for models to be ready (timeout: 300s)... llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated condition met llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium condition met llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated condition met ✅ Simulator models ready Waiting for governed MaaSModelRefs to be Ready (timeout: 300s)... ✅ Governed MaaSModelRefs ready Waiting for MaaS Kuadrant AuthPolicies to be enforced (selector: app.kubernetes.io/managed-by=maas-controller, timeout: 180s)... ✅ All MaaS AuthPolicies enforced (1 policies) 2026-09-19T01:53:28Z deploy_models end Patching Authorino to log_level DEBUG... authorino.operator.authorino.kuadrant.io/authorino patched ✅ Authorino patched to log_level DEBUG Waiting for deployment spec update to be observed... Waiting for deployment spec update to be observed... Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out ---------------------------------------- Deploying ai-gateway-controller ---------------------------------------- 2026-09-19T01:53:30Z deploy_ai_gateway_controller start Deploying ai-gateway-controller image: quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-37 praxis-extproc image: quay.io/opendatahub/odh-praxis-extproc:odh-stable namespace: opendatahub gateway: openshift-ingress/maas-default-gateway remove maas IPP: true serviceaccount/ai-gateway-controller created clusterrole.rbac.authorization.k8s.io/ai-gateway-controller-role created clusterrolebinding.rbac.authorization.k8s.io/ai-gateway-controller-rolebinding created configmap/ai-gateway-controller-parameters created deployment.apps/ai-gateway-controller created Waiting for deployment "ai-gateway-controller" rollout to finish: 0 of 1 updated replicas are available... deployment "ai-gateway-controller" successfully rolled out Triggering immediate praxis-extproc install ... deployment.apps/ai-gateway-controller restarted Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... deployment "ai-gateway-controller" successfully rolled out Annotating MaasTenantConfig models-as-a-service/default-tenant: type=praxis status=cleanup-complete ... maastenantconfig.maas.opendatahub.io/default-tenant annotated ERROR: payload-processing-type did not stick (got '')