Fetching https://github.com/opendatahub-io/models-as-a-service@7879675aa66d002425c7b32375d6d0156eaa0c56 into /workspace/source/test/maas-e2e ... MaaS checkout at: 7879675aa66d002425c7b32375d6d0156eaa0c56 Patched /workspace/source/test/maas-e2e/scripts/deploy.sh Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS checkout fixture root) WARN: deploy-models.sh AuthPolicy wait block not found; patch manually Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS AuthPolicy wait scope) NOTE: rewriting AI_GATEWAY_CONTROLLER_IMAGE from snapshot digest to tag odh-pr-177 MaaS platform images: MAAS_API_IMAGE=quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 MAAS_CONTROLLER_IMAGE=quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 AI_GATEWAY_CONTROLLER_IMAGE=quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177 PRAXIS_EXTPROC_IMAGE=quay.io/opendatahub/odh-praxis-extproc:odh-stable ---------------------------------------- ai-gateway-controller E2E on OpenShift ---------------------------------------- Checking prerequisites... Prerequisites met — logged in as: system:admin DEPLOY_MODE: kustomize MAAS_API_IMAGE: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 MAAS_CONTROLLER_IMAGE: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 AI_GATEWAY_CONTROLLER_IMAGE: quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177 PRAXIS_EXTPROC_IMAGE: quay.io/opendatahub/odh-praxis-extproc:odh-stable 2026-10-31T00:46:16Z deploy_platform start Deploying MaaS platform via ODH operator... Gateway ingress mode for deploy.sh: ocproute (loadbalancer | ocproute) Using custom MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 Using custom MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 Deployment mode: kustomize Using policy engine: rhcl (Authorino namespace: kuadrant-system) [INFO] =================================================== [INFO] Models-as-a-Service Deployment [INFO] =================================================== [INFO] Validating configuration... [INFO] Configuration validated successfully [INFO] Deployment configuration: [INFO] Mode: kustomize [INFO] Operator: odh [INFO] Policy Engine: rhcl [INFO] Namespace: opendatahub [INFO] TLS Backend: true [INFO] External OIDC: false [INFO] MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] Installing shared dependencies for kustomize mode... [INFO] =================================================== [INFO] Setup Shared Dependencies (Helm) [INFO] =================================================== [INFO] Operator type: odh [INFO] Deployment mode: kustomize [INFO] Ingress mode: ocproute [INFO] Policy engine: rhcl [INFO] Chart source: https://github.com/opendatahub-io/odh-gitops.git @ main (fadfb31e89ae898b04a9a75bfa8bb91588d7e625) [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] Payload processing image: chart default [INFO] AI Gateway operator image: catalog default [INFO] helm not found, installing v3.17.3... [INFO] helm installed: v3.17.3+ge4da497 [INFO] Cloning odh-gitops chart (branch: main, commit: fadfb31e89ae898b04a9a75bfa8bb91588d7e625)... Cloning into '/tmp/tmp.trpDbx7N4W/odh-gitops'... From https://github.com/opendatahub-io/odh-gitops * branch fadfb31e89ae898b04a9a75bfa8bb91588d7e625 -> FETCH_HEAD [INFO] Chart resolved at: /tmp/tmp.trpDbx7N4W/odh-gitops/charts/rhai-on-openshift-chart [INFO] [INFO] Phase 1: Installing operators via Helm chart... [INFO] Running: helm upgrade --install rhoai-deps Release "rhoai-deps" does not exist. Installing it now. NAME: rhoai-deps LAST DEPLOYED: Sat Oct 31 00:46:21 2026 NAMESPACE: rhoai-deps STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: ODH Chart installed successfully! === Installation Summary === Operator: ODH Namespace: rhoai-deps Profile: maas === Enabled Components === - aigateway: Removed - aipipelines: Removed - dashboard: Removed - feastoperator: Removed - kserve: Managed - kueue: Removed - mlflowoperator: Removed - modelregistry: Removed - ogx: Removed - ray: Removed - sparkoperator: Removed - trainer: Removed - trainingoperator: Removed - trustyai: Removed - workbenches: Removed === Enabled Services === - monitoring: Removed === Enabled Dependencies === - certManager - jobSet - rhcl === Next Steps === 1. Wait for operators to be ready: kubectl get csv -A | grep -E "(rhods|opendatahub|cert-manager|leader-worker)" 2. If CRs were not created (first install), run the same helm upgrade command again. 3. Check DataScienceCluster status: kubectl get datasciencecluster default-dsc -o jsonpath='{.status.phase}' === Enable Authorino TLS (Optional) === To enable TLS for Authorino: 1. Annotate the service to trigger TLS certificate generation: kubectl annotate svc/authorino-authorino-authorization \ service.beta.openshift.io/serving-cert-secret-name=authorino-server-cert \ -n kuadrant-system 2. Patch the Authorino CR to enable TLS: kubectl patch authorino/authorino -n kuadrant-system --type=merge -p '{"spec":{"clusterWide":true,"listener":{"tls":{"enabled":true,"certSecretRef":{"name":"authorino-server-cert"}}},"oidcServer":{"tls":{"enabled":false}}}}' === Documentation === For more information, see the chart README: https://github.com/opendatahub-io/odh-gitops/tree/main/charts/rhai-on-openshift-chart [INFO] Enabling bundle unpack retry on OperatorGroup cert-manager-operator/openshift-cert-manager-operator operatorgroup.operators.coreos.com/openshift-cert-manager-operator annotated [INFO] Enabling bundle unpack retry on OperatorGroup kuadrant-system/rhcl-operator operatorgroup.operators.coreos.com/rhcl-operator annotated [INFO] Enabling bundle unpack retry on OperatorGroup opendatahub-operator-system/opendatahub-operator operatorgroup.operators.coreos.com/opendatahub-operator annotated [INFO] Enabling bundle unpack retry on OperatorGroup openshift-jobset-operator/job-set operatorgroup.operators.coreos.com/job-set annotated [INFO] [INFO] Waiting for OLM to unpack operator bundles... [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator opendatahub-operator-system/opendatahub-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator opendatahub-operator-system/opendatahub-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator opendatahub-operator-system/opendatahub-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator (2000s timeout) [INFO] Operator bundles unpacked [INFO] [INFO] Waiting for operator CRDs to be established... ⏳ Waiting for CRD dscinitializations.dscinitialization.opendatahub.io to appear (timeout: 600s)… ✅ CRD dscinitializations.dscinitialization.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io condition met ⏳ Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 600s)… ✅ CRD datascienceclusters.datasciencecluster.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met ⏳ Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 600s)… ✅ CRD kuadrants.kuadrant.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [INFO] [INFO] Waiting for ODH operator to be ready (webhook must be serving)... [INFO] Operator CSV is Succeeded in opendatahub-operator-system deployment.apps/opendatahub-operator-controller-manager condition met [INFO] [INFO] Applying local MaaS CRDs for kustomize mode... ⏳ Applying MaaS Controller CRDs from /workspace/source/test/maas-e2e/deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io created ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] [INFO] Phase 2: Applying CRD-dependent resources (DSC, DSCI, Kuadrant CR)... [INFO] Running: helm upgrade --install rhoai-deps Release "rhoai-deps" has been upgraded. Happy Helming! NAME: rhoai-deps LAST DEPLOYED: Sat Oct 31 00:47:33 2026 NAMESPACE: rhoai-deps STATUS: deployed REVISION: 2 TEST SUITE: None NOTES: ODH Chart installed successfully! === Installation Summary === Operator: ODH Namespace: rhoai-deps Profile: maas === Enabled Components === - aigateway: Removed - aipipelines: Removed - dashboard: Removed - feastoperator: Removed - kserve: Managed - kueue: Removed - mlflowoperator: Removed - modelregistry: Removed - ogx: Removed - ray: Removed - sparkoperator: Removed - trainer: Removed - trainingoperator: Removed - trustyai: Removed - workbenches: Removed === Enabled Services === - monitoring: Removed === Enabled Dependencies === - certManager - jobSet - rhcl === Next Steps === 1. Wait for operators to be ready: kubectl get csv -A | grep -E "(rhods|opendatahub|cert-manager|leader-worker)" 2. If CRs were not created (first install), run the same helm upgrade command again. 3. Check DataScienceCluster status: kubectl get datasciencecluster default-dsc -o jsonpath='{.status.phase}' === Enable Authorino TLS (Optional) === To enable TLS for Authorino: 1. Annotate the service to trigger TLS certificate generation: kubectl annotate svc/authorino-authorino-authorization \ service.beta.openshift.io/serving-cert-secret-name=authorino-server-cert \ -n kuadrant-system 2. Patch the Authorino CR to enable TLS: kubectl patch authorino/authorino -n kuadrant-system --type=merge -p '{"spec":{"clusterWide":true,"listener":{"tls":{"enabled":true,"certSecretRef":{"name":"authorino-server-cert"}}},"oidcServer":{"tls":{"enabled":false}}}}' === Documentation === For more information, see the chart README: https://github.com/opendatahub-io/odh-gitops/tree/main/charts/rhai-on-openshift-chart [INFO] Waiting for Kuadrant policy engine to become ready... [INFO] Waiting for: Kuadrant ready in kuadrant-system (timeout: 600s) [INFO] Kuadrant ready in kuadrant-system - Ready [INFO] [INFO] Shared dependencies installed successfully [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [WARN] DEPLOYING POC POSTGRESQL — NOT INTENDED FOR PRODUCTION USE [WARN] Data is stored in ephemeral storage and will be lost on pod restart. [WARN] For production, use --postgres-connection with an external database [WARN] (AWS RDS, Crunchy Operator, Azure Database, etc.) [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ⚠️ WARNING FOR PRODUCTION USE. ⚠️ ┃ ┃ This deploys PostgreSQL with ephemeral storage (emptyDir). ┃ ┃ Data WILL be lost on pod restart. ┃ ┃ For production, use an external database: ┃ ┃ deploy.sh --postgres-connection postgresql://... ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 🔧 Fresh install: Deploying PostgreSQL in infrastructure namespace 'odh-ai-gateway-infra'... 📦 Creating infrastructure namespace 'odh-ai-gateway-infra'... namespace/odh-ai-gateway-infra created Generated random PostgreSQL password (stored in secret postgres-creds) Creating PostgreSQL deployment... ⚠️ Using POC configuration (ephemeral storage) Using default PostgreSQL image (operator CSV not available) Image: registry.redhat.io/rhel9/postgresql-16:latest secret/postgres-creds created deployment.apps/postgres created service/postgres created secret/maas-db-config created Waiting for PostgreSQL to be ready... deployment.apps/postgres condition met ✅ PostgreSQL deployed successfully Namespace: odh-ai-gateway-infra Database: maas User: maas Secret: maas-db-config (contains DB_CONNECTION_URL with FQDN) ⚠️ For production, use AWS RDS, Crunchy Operator, or Azure Database Note: Schema migrations run automatically when maas-api starts [INFO] Shared dependencies installed [INFO] [INFO] MaaS Controller... [INFO] Phase 1: Applying MaaS CRDs and waiting until Established (controller creates Config after CRD is ready)... ⏳ Applying MaaS Controller CRDs from /workspace/source/test/maas-e2e/scripts/../deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io unchanged customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io unchanged ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] Phase 2: Applying full controller kustomize (same as operator: deployment/base/maas-controller/default)... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io configured serviceaccount/maas-controller created role.rbac.authorization.k8s.io/maas-controller-leader-election-role created clusterrole.rbac.authorization.k8s.io/maas-controller-cluster-config-role created clusterrole.rbac.authorization.k8s.io/maas-controller-ocp-role created clusterrole.rbac.authorization.k8s.io/maas-controller-role created clusterrole.rbac.authorization.k8s.io/maas-owner-role created clusterrole.rbac.authorization.k8s.io/maas-viewer-role created rolebinding.rbac.authorization.k8s.io/maas-controller-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-cluster-config-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-ocp-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-rolebinding created configmap/maas-parameters created service/maas-controller-metrics created service/maas-controller-webhook-service created deployment.apps/maas-controller created servicemonitor.monitoring.coreos.com/maas-controller-metrics created networkpolicy.networking.k8s.io/maas-controller-allow-monitoring created validatingwebhookconfiguration.admissionregistration.k8s.io/maas-validating-webhook-configuration created [INFO] Restarting maas-controller to pick up configuration changes... deployment.apps/maas-controller restarted [INFO] Waiting for maas-controller to be ready... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... deployment "maas-controller" successfully rolled out [INFO] Controller ready. [INFO] [INFO] Waiting for Tenant reconciler to deploy maas-api... [INFO] Applying secret migration RBAC to namespace odh-ai-gateway-infra... role.rbac.authorization.k8s.io/maas-controller-secret-migrate created rolebinding.rbac.authorization.k8s.io/maas-controller-secret-migrate created [INFO] maas-api deployment found in odh-ai-gateway-infra, waiting for rollout... Waiting for deployment "maas-api" rollout to finish: 0 of 1 updated replicas are available... deployment "maas-api" successfully rolled out [INFO] maas-api is ready [INFO] Configuring TLS backend for Authorino and MaaS API... * Waiting for deployment/authorino in kuadrant-system (timeout: 300s)... * Found deployment/authorino [INFO] Running TLS configuration script... [INFO] TLS configuration script completed successfully [INFO] Restarting deployments to pick up TLS configuration... deployment.apps/maas-api restarted deployment.apps/authorino restarted [INFO] Waiting for Authorino deployment to be ready... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out [INFO] TLS backend configuration complete [INFO] [INFO] MaaS API and MaaS Controller deployment completed successfully! [INFO] maas-api image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 (namespace: odh-ai-gateway-infra) [INFO] maas-controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 (namespace: opendatahub) [INFO] =================================================== [INFO] Models-as-a-Service Deployment completed successfully! [INFO] =================================================== * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: (informational only), AIGatewayReady: False * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' Raising llmisvc-controller-manager CPU in opendatahub (request 1, limit 2)... deployment.apps/llmisvc-controller-manager resource requirements updated Waiting for deployment "llmisvc-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "llmisvc-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "llmisvc-controller-manager" rollout to finish: 1 old replicas are pending termination... deployment "llmisvc-controller-manager" successfully rolled out llmisvc-controller-manager resources: {"limits":{"cpu":"2","memory":"300Mi"},"requests":{"cpu":"1","memory":"300Mi"}} ⚠️ WARNING: Skipping Authorino readiness check (SKIP_AUTH_CHECK=true) ✅ MaaS platform deployment completed 2026-10-31T00:50:54Z deploy_platform end ---------------------------------------- Deploying Models ---------------------------------------- 2026-10-31T00:50:54Z deploy_models start Deploying MaaS system (free + premium: LLMIS + MaaSModelRef + MaaSAuthPolicy + MaaSSubscription) Waiting for Gateway openshift-ingress/maas-default-gateway to be Programmed=True (timeout: 600s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met ✅ Gateway openshift-ingress/maas-default-gateway is Programmed Creating 'llm' namespace... namespace/llm created 'models-as-a-service' namespace already exists Waiting for ready endpoints on maas-controller-webhook-service in opendatahub (timeout: 120s)... ✅ MaaS admission webhook has ready endpoints: 10.133.0.27 maasauthpolicy.maas.opendatahub.io/premium-simulator-access created maasauthpolicy.maas.opendatahub.io/simulator-access created maasmodelref.maas.opendatahub.io/e2e-distinct-2-simulated created maasmodelref.maas.opendatahub.io/e2e-distinct-simulated created maasmodelref.maas.opendatahub.io/e2e-embedding-simulated created maasmodelref.maas.opendatahub.io/e2e-trlp-test-simulated created maasmodelref.maas.opendatahub.io/e2e-unconfigured-facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/echo-simulator created maasmodelref.maas.opendatahub.io/facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/premium-simulated-simulated-premium created maassubscription.maas.opendatahub.io/premium-simulator-subscription created maassubscription.maas.opendatahub.io/simulator-subscription created llminferenceservice.serving.kserve.io/e2e-distinct-2-simulated created llminferenceservice.serving.kserve.io/e2e-distinct-simulated created llminferenceservice.serving.kserve.io/e2e-embedding-simulated created llminferenceservice.serving.kserve.io/e2e-trlp-test-simulated created llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/echo-simulator created llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium created ✅ MaaS system deployed (free + premium + e2e test fixtures) Waiting for models to be ready (timeout: 300s)... llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated condition met llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium condition met llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated condition met ✅ Simulator models ready Waiting for governed MaaSModelRefs to be Ready (timeout: 300s per model)... maasmodelref.maas.opendatahub.io/facebook-opt-125m-simulated condition met maasmodelref.maas.opendatahub.io/premium-simulated-simulated-premium condition met ✅ Governed MaaSModelRefs ready Waiting for AuthPolicies targeting gateway 'maas-default-gateway' to be enforced (timeout: 180s)... Skipping openshift-ai-inference-authn (targets openshift-ai-inference, not maas-default-gateway) ✅ All AuthPolicies targeting maas-default-gateway enforced (1 policies) 2026-10-31T00:52:11Z deploy_models end Patching Authorino to log_level DEBUG... authorino.operator.authorino.kuadrant.io/authorino patched ✅ Authorino patched to log_level DEBUG deployment "authorino" successfully rolled out ---------------------------------------- Deploying ai-gateway-controller ---------------------------------------- 2026-10-31T00:52:12Z deploy_ai_gateway_controller start Deploying ai-gateway-controller image: quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-177 praxis-extproc image: quay.io/opendatahub/odh-praxis-extproc:odh-stable namespace: opendatahub gateway: openshift-ingress/maas-default-gateway customresourcedefinition.apiextensions.k8s.io/aiguardrails.aigateway.opendatahub.io created serviceaccount/ai-gateway-controller created clusterrole.rbac.authorization.k8s.io/ai-gateway-controller-role created clusterrolebinding.rbac.authorization.k8s.io/ai-gateway-controller-rolebinding created configmap/ai-gateway-controller-parameters created service/ai-gateway-controller-metrics created deployment.apps/ai-gateway-controller created servicemonitor.monitoring.coreos.com/ai-gateway-controller-metrics created networkpolicy.networking.k8s.io/ai-gateway-controller-allow-metrics-traffic created Waiting for deployment "ai-gateway-controller" rollout to finish: 0 of 1 updated replicas are available... deployment "ai-gateway-controller" successfully rolled out Triggering immediate praxis-extproc reconcile ... deployment.apps/ai-gateway-controller restarted Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... deployment "ai-gateway-controller" successfully rolled out Waiting for shared praxis-extproc (quay.io/opendatahub/odh-praxis-extproc:odh-stable) in openshift-ingress: payload-processing and payload-pre-processing (timeout: 300s) ... ✅ shared praxis-extproc ready: openshift-ingress/payload-processing and openshift-ingress/payload-pre-processing Annotating praxis IPP resources opendatahub.io/managed=false so maas-controller skips them ... deployment.apps/payload-processing annotated service/payload-processing annotated destinationrule.networking.istio.io/payload-processing annotated deployment.apps/payload-pre-processing annotated service/payload-pre-processing annotated destinationrule.networking.istio.io/payload-pre-processing annotated envoyfilter.networking.istio.io/payload-processing annotated networkpolicy.networking.k8s.io/payload-processing annotated Verified shared ext_proc dataplane image: openshift-ingress/payload-processing=quay.io/opendatahub/odh-praxis-extproc:odh-stable ai-gateway-controller rollout complete 2026-10-31T00:52:50Z deploy_ai_gateway_controller end ---------------------------------------- Enabling tenant namespace discovery ---------------------------------------- 2026-10-31T00:52:50Z tenant_namespace_discovery start Enabling --enable-tenant-namespace-discovery on maas-controller... maas-controller already has tenant namespace discovery enabled 2026-10-31T00:53:01Z tenant_namespace_discovery end ---------------------------------------- Setting up variables for tests ---------------------------------------- -- Setting up variables for tests -- HOST: maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev MAAS_API_BASE_URL: https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/maas-api ---------------------------------------- Setting up test tokens ---------------------------------------- Setting up premium test token (SA-based, works when oc whoami -t is unavailable)... Creating namespace: premium-users-namespace namespace/premium-users-namespace created Creating service account: premium-service-account serviceaccount/premium-service-account created Patching MaaSAuthPolicy premium-simulator-access to include system:serviceaccount:premium-users-namespace:premium-service-account... maasauthpolicy.maas.opendatahub.io/premium-simulator-access patched Patching MaaSSubscription premium-simulator-subscription to include system:serviceaccount:premium-users-namespace:premium-service-account... Error from server (InternalError): Internal error occurred: failed calling webhook "vmaassubscription.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maassubscription?timeout=10s": EOF ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Saved to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs Saved 9 resource type(s) to /workspace/artifacts-dir/maas-crs Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 9 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Collecting pod logs from namespace odh-ai-gateway-infra to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Saved 3 pod log file(s) to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Collecting pod logs from namespace models-as-a-service to /workspace/artifacts-dir/pod-logs/models-as-a-service Saved 0 0 pod log file(s) to /workspace/artifacts-dir/pod-logs/models-as-a-service Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Collecting pod logs from namespace kuadrant-system to /workspace/artifacts-dir/pod-logs/kuadrant-system Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/kuadrant-system Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 13 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Collecting pod logs from namespace llm to /workspace/artifacts-dir/pod-logs/llm Saved 8 pod log file(s) to /workspace/artifacts-dir/pod-logs/llm Skipping namespace istio-system (not found) ============================================== ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Saved to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs Saved 10 resource type(s) to /workspace/artifacts-dir/maas-crs Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 9 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Collecting pod logs from namespace odh-ai-gateway-infra to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Saved 3 pod log file(s) to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Collecting pod logs from namespace models-as-a-service to /workspace/artifacts-dir/pod-logs/models-as-a-service Saved 0 0 pod log file(s) to /workspace/artifacts-dir/pod-logs/models-as-a-service Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Collecting pod logs from namespace kuadrant-system to /workspace/artifacts-dir/pod-logs/kuadrant-system Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/kuadrant-system Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 13 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Collecting pod logs from namespace llm to /workspace/artifacts-dir/pod-logs/llm Saved 8 pod log file(s) to /workspace/artifacts-dir/pod-logs/llm Skipping namespace istio-system (not found) ============================================== Collecting AIGC cluster CR extensions to /workspace/artifacts-dir/maas-crs Saved datascienceclusters.yaml Saved httproutes.yaml Saved gateways.yaml Collecting MaaS diagnostics to /workspace/artifacts-dir/gather-maas (log: /workspace/artifacts-dir/maas-must-gather.log) ... MaaS must-gather complete (see /workspace/artifacts-dir/maas-must-gather.log) ======================================== Cluster / Namespace Info ======================================== --- Current context --- admin --- Logged-in user --- system:admin --- Cluster domain --- apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev DEPLOYMENT_NAMESPACE: opendatahub MAAS_API_DEPLOYMENT_NAMESPACE: odh-ai-gateway-infra MAAS_SUBSCRIPTION_NAMESPACE: models-as-a-service AUTHORINO_NAMESPACE: kuadrant-system ======================================== MaaS API Deployment ======================================== --- maas-api pods --- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES maas-api-bbf56748-r8f68 1/1 Running 1 (5m8s ago) 5m20s 10.133.0.33 ip-10-0-134-13.us-west-1.compute.internal --- maas-api service --- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR maas-api ClusterIP 172.31.192.24 8443/TCP 5m42s app.kubernetes.io/component=api,app.kubernetes.io/name=maas-api,app.kubernetes.io/part-of=models-as-a-service,maas.opendatahub.io/tenant-instance=maas-api ======================================== maas-controller ======================================== --- maas-controller pods --- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES maas-controller-748b6794c7-vvvsz 1/1 Running 1 (3m46s ago) 6m22s 10.133.0.27 ip-10-0-134-13.us-west-1.compute.internal --- maas-controller namespace env --- INFRA_NAMESPACE=AUTO ======================================== Kuadrant Policies ======================================== --- AuthPolicies (all namespaces) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE openshift-ingress maas-gateway-auth True True Gateway maas-default-gateway 6m17s openshift-ingress openshift-ai-inference-authn True False Gateway openshift-ai-inference 5m49s --- TokenRateLimitPolicies (all namespaces) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE llm maas-trlp-facebook-opt-125m-simulated True True HTTPRoute facebook-opt-125m-simulated-kserve-route 3m26s llm maas-trlp-premium-simulated-simulated-premium True True HTTPRoute premium-simulated-simulated-premium-kserve-route 3m24s openshift-ingress gateway-default-deny True False Gateway maas-default-gateway 5m39s ======================================== MaaS CRs (maas.opendatahub.io) ======================================== --- Configs (cluster-scoped) --- NAME READY AGE default True 6m17s --- AITenants (all namespaces) --- NAMESPACE NAME READY TENANT NAMESPACE GATEWAY AGE ai-tenants models-as-a-service True models-as-a-service maas-default-gateway 5m47s --- Tenants (all namespaces) --- --- MaaSAuthPolicies --- NAME PHASE AGE AUTHPOLICIES premium-simulator-access Active 4m3s simulator-access Active 4m3s --- MaaSSubscriptions --- NAME PHASE PRIORITY AGE premium-simulator-subscription Active 20 4m2s simulator-subscription Active 10 4m2s --- MaaSSubscription status details --- premium-simulator-subscription: Active - successfully reconciled simulator-subscription: Active - successfully reconciled --- MaaSModelRefs (all namespaces) --- NAMESPACE NAME PHASE ENDPOINT HTTPROUTE GATEWAY AGE llm e2e-distinct-2-simulated Pending e2e-distinct-2-simulated-kserve-route maas-default-gateway 4m3s llm e2e-distinct-simulated Pending e2e-distinct-simulated-kserve-route maas-default-gateway 4m3s llm e2e-embedding-simulated Pending e2e-embedding-simulated-kserve-route maas-default-gateway 4m3s llm e2e-trlp-test-simulated Pending e2e-trlp-test-simulated-kserve-route maas-default-gateway 4m2s llm e2e-unconfigured-facebook-opt-125m-simulated Pending e2e-unconfigured-facebook-opt-125m-simulated-kserve-route maas-default-gateway 4m2s llm echo-simulator Pending echo-simulator-kserve-route maas-default-gateway 4m2s llm facebook-opt-125m-simulated Ready https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/ facebook-opt-125m-simulated-kserve-route maas-default-gateway 4m2s llm premium-simulated-simulated-premium Ready https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/ premium-simulated-simulated-premium-kserve-route maas-default-gateway 4m2s --- MaasTenantConfigs --- NAME READY REASON AGE INFRANAMESPACE default-tenant True Reconciled 5m45s odh-ai-gateway-infra --- MaasTenantConfig status details --- default-tenant: True - MaaS platform manifests applied and maas-api deployment is available --- ExternalModels (maas group) --- ======================================== Inference CRs (inference.opendatahub.io) ======================================== --- ExternalModels (inference group) --- --- ExternalProviders (inference group) --- ======================================== AI Gateway CRs ======================================== --- AIGateways (cluster-scoped) --- --- LLMBatchGateways (all namespaces) --- ======================================== KServe Resources ======================================== --- LLMInferenceServices (all namespaces) --- NAMESPACE NAME URL READY REASON AGE URLS llm e2e-distinct-2-simulated https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/llm/e2e-distinct-2-simulated True 4m3s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-distinct-simulated https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/llm/e2e-distinct-simulated True 4m2s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-embedding-simulated https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/llm/e2e-embedding-simulated True 4m2s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-trlp-test-simulated https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/llm/e2e-trlp-test-simulated True 4m1s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm e2e-unconfigured-facebook-opt-125m-simulated https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/llm/e2e-unconfigured-facebook-opt-125m-simulated True 4m1s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm echo-simulator https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/llm/echo-simulator True 4m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm facebook-opt-125m-simulated https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/llm/facebook-opt-125m-simulated True 4m https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ llm premium-simulated-simulated-premium https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/llm/premium-simulated-simulated-premium True 3m59s https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ ======================================== Kuadrant Policies (managed by maas-controller) ======================================== --- AuthPolicies (managed) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE openshift-ingress maas-gateway-auth True True Gateway maas-default-gateway 6m20s --- TokenRateLimitPolicies (managed) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE llm maas-trlp-facebook-opt-125m-simulated True True HTTPRoute facebook-opt-125m-simulated-kserve-route 3m39s llm maas-trlp-premium-simulated-simulated-premium True True HTTPRoute premium-simulated-simulated-premium-kserve-route 3m37s openshift-ingress gateway-default-deny True False Gateway maas-default-gateway 5m52s ======================================== Payload Processing / IPP ======================================== --- IPP Deployments (openshift-ingress) --- --- IPP Services (openshift-ingress) --- --- EnvoyFilters (openshift-ingress) --- NAME AGE data-science-authn-filter 7m44s kuadrant-auth-maas-default-gateway 5m51s kuadrant-maas-default-gateway 5m51s kuadrant-ratelimiting-maas-default-gateway 5m51s maas-default-gateway-authn-ssl 4m13s maas-model-access-logs 2m56s payload-processing 2m59s --- IPP ConfigMaps (openshift-ingress) --- ======================================== NetworkPolicies ======================================== --- NetworkPolicies (openshift-ingress) --- NAME POD-SELECTOR AGE data-science-gateway-allow gateway.networking.k8s.io/gateway-name=data-science-gateway 7m55s istiod-allow app=istiod 9m10s kube-auth-proxy app=kube-auth-proxy 7m53s maas-default-gateway-allow gateway.networking.k8s.io/gateway-name=maas-default-gateway 9m10s openshift-ai-inference-allow gateway.networking.k8s.io/gateway-name=openshift-ai-inference 9m10s openshift-ingress-deny-all 19m payload-processing maas.opendatahub.io/tenant-instance in (payload-pre-processing,payload-processing) 3m9s router-default ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default 19m ======================================== Test User Information ======================================== No test token available (not logged in via oc) ======================================== Subscription → Model Mapping ======================================== Subscription: premium-simulator-subscription Owner users: (none) Owner groups: premium-user Models: llm/premium-simulated-simulated-premium Subscription: simulator-subscription Owner users: (none) Owner groups: system:authenticated Models: llm/facebook-opt-125m-simulated ======================================== Available Models (MaaSModelRefs) ======================================== Model Reference → Model ID / Endpoint llm/e2e-distinct-2-simulated → e2e-distinct-2-simulated (Pending) llm/e2e-distinct-simulated → e2e-distinct-simulated (Pending) llm/e2e-embedding-simulated → e2e-embedding-simulated (Pending) llm/e2e-trlp-test-simulated → e2e-trlp-test-simulated (Pending) llm/e2e-unconfigured-facebook-opt-125m-simulated → e2e-unconfigured-facebook-opt-125m-simulated (Pending) llm/echo-simulator → echo-simulator (Pending) llm/facebook-opt-125m-simulated → facebook-opt-125m-simulated (Ready) Endpoint: https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/ llm/premium-simulated-simulated-premium → premium-simulated-simulated-premium (Ready) Endpoint: https://maas.apps.ea194ad028ecb32f19bb.prod.konflux-ocp-ci.dev/ ======================================== Gateway / HTTPRoutes ======================================== --- Gateway --- NAME CLASS ADDRESS PROGRAMMED AGE maas-default-gateway openshift-default maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local True 9m11s --- HTTPRoutes (maas-api) --- NAME HOSTNAMES AGE maas-api-route 6m6s ======================================== Authorino ======================================== --- Authorino pods --- --- ======================================== Subscription Selector Endpoint Validation ======================================== Expected URL (from resolved maas-api deployment namespace): https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select (MAAS_API_DEPLOYMENT_NAMESPACE resolved to: odh-ai-gateway-infra) --- Sample AuthPolicy subscription-info configuration --- Inspecting: openshift-ingress/maas-gateway-auth Actual URL in AuthPolicy: https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select ✅ Request body includes requestedModel field Model reference: / Request body preview: { "groups": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.groups : (has(auth.identity.iss) ? (has(auth.identity.groups) && auth.identity.groups.all(g, g.matches('^[A-Za-z0-9:#._/ -]+$')) ? auth.identity.groups : []) : (has(auth.identity.user.groups) && auth.identity.user.groups.all(g, g.matches('^[A-Za-z0-9:#._/ -]+$')) ? auth.identity.user.groups : [])), "username": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.username : (has(auth.identity.preferred_username) ? auth.identity.preferred_username : (has(auth.identity.sub) ? auth.identity.sub : auth.identity.user.username)), "requestedSubscription": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.subscription : ("x-maas-subscription" in request.headers ? request.headers["x-maas-subscription"] : ""), "requestedModel": (size(request.path.split("/").filter(x, x != "")) >= 2 && request.path.split("/").filter(x, x != "")[0] != "v1" && request.path.split("/").filter(x, x != "")[0] != "maas-api" ? request.path.split("/").filter(x, x != "")[0] + "/" + request.path.split("/").filter(x, x != "")[1] : ("x-gateway-model-name" in request.headers ? request.headers["x-gateway-model-name"] : "")) --- Connectivity test (from kuadrant-system, simulates Authorino) --- curl -vsk -m 10 -X POST 'https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select' -H 'Content-Type: application/json' -d '{}' kubectl run failed or timed out ======================================== DNS Resolution Check ======================================== Resolving: maas-api.odh-ai-gateway-infra.svc.cluster.local nslookup failed ======================================== Configuration Summary ======================================== This summary helps compare local vs CI runs: AITenants (all namespaces): 1 MaaSModelRefs (all namespaces): 8 MaaSSubscriptions (models-as-a-service): 2 MaaSAuthPolicies (models-as-a-service): 2 Generated Kuadrant AuthPolicies: 1 Generated Kuadrant TokenRateLimitPolicies: 3 LLMInferenceServices (all namespaces): 8 ExternalModels - inference (all namespaces): 0 ExternalProviders - inference (all namespaces): 0 EnvoyFilters (openshift-ingress): 7 IPP Deployments (openshift-ingress): 0 NetworkPolicies (openshift-ingress): 8 Subscription selector URL: https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select Test user: system:admin