Fetching https://github.com/opendatahub-io/models-as-a-service@7879675aa66d002425c7b32375d6d0156eaa0c56 into /workspace/source/test/maas-e2e ... MaaS checkout at: 7879675aa66d002425c7b32375d6d0156eaa0c56 Patched /workspace/source/test/maas-e2e/scripts/deploy.sh Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS checkout fixture root) NOTE: rewriting AI_GATEWAY_CONTROLLER_IMAGE from snapshot digest to tag odh-pr-185 MaaS platform images: MAAS_API_IMAGE=quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 MAAS_CONTROLLER_IMAGE=quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 AI_GATEWAY_CONTROLLER_IMAGE=quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-185 PRAXIS_EXTPROC_IMAGE=quay.io/opendatahub/odh-praxis-extproc:odh-stable ---------------------------------------- ai-gateway-controller E2E on OpenShift ---------------------------------------- Checking prerequisites... Prerequisites met — logged in as: system:admin DEPLOY_MODE: kustomize MAAS_API_IMAGE: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 MAAS_CONTROLLER_IMAGE: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 AI_GATEWAY_CONTROLLER_IMAGE: quay.io/opendatahub/odh-ai-gateway-controller:odh-pr-185 PRAXIS_EXTPROC_IMAGE: quay.io/opendatahub/odh-praxis-extproc:odh-stable 2026-11-02T18:06:06Z deploy_platform start Deploying MaaS platform via ODH operator... Gateway ingress mode for deploy.sh: ocproute (loadbalancer | ocproute) Using custom MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 Using custom MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 Deployment mode: kustomize Using policy engine: rhcl (Authorino namespace: kuadrant-system) [INFO] =================================================== [INFO] Models-as-a-Service Deployment [INFO] =================================================== [INFO] Validating configuration... [INFO] Configuration validated successfully [INFO] Deployment configuration: [INFO] Mode: kustomize [INFO] Operator: odh [INFO] Policy Engine: rhcl [INFO] Namespace: opendatahub [INFO] TLS Backend: true [INFO] External OIDC: false [INFO] MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] Installing shared dependencies for kustomize mode... [INFO] =================================================== [INFO] Setup Shared Dependencies (Helm) [INFO] =================================================== [INFO] Operator type: odh [INFO] Deployment mode: kustomize [INFO] Ingress mode: ocproute [INFO] Policy engine: rhcl [INFO] Chart source: https://github.com/opendatahub-io/odh-gitops.git @ main (fadfb31e89ae898b04a9a75bfa8bb91588d7e625) [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 [INFO] Payload processing image: chart default [INFO] AI Gateway operator image: catalog default [INFO] helm not found, installing v3.17.3... [INFO] helm installed: v3.17.3+ge4da497 [INFO] Cloning odh-gitops chart (branch: main, commit: fadfb31e89ae898b04a9a75bfa8bb91588d7e625)... Cloning into '/tmp/tmp.XLCobZS4tU/odh-gitops'... From https://github.com/opendatahub-io/odh-gitops * branch fadfb31e89ae898b04a9a75bfa8bb91588d7e625 -> FETCH_HEAD [INFO] Chart resolved at: /tmp/tmp.XLCobZS4tU/odh-gitops/charts/rhai-on-openshift-chart [INFO] [INFO] Phase 1: Installing operators via Helm chart... [INFO] Running: helm upgrade --install rhoai-deps Release "rhoai-deps" does not exist. Installing it now. NAME: rhoai-deps LAST DEPLOYED: Mon Nov 2 18:06:11 2026 NAMESPACE: rhoai-deps STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: ODH Chart installed successfully! === Installation Summary === Operator: ODH Namespace: rhoai-deps Profile: maas === Enabled Components === - aigateway: Removed - aipipelines: Removed - dashboard: Removed - feastoperator: Removed - kserve: Managed - kueue: Removed - mlflowoperator: Removed - modelregistry: Removed - ogx: Removed - ray: Removed - sparkoperator: Removed - trainer: Removed - trainingoperator: Removed - trustyai: Removed - workbenches: Removed === Enabled Services === - monitoring: Removed === Enabled Dependencies === - certManager - jobSet - rhcl === Next Steps === 1. Wait for operators to be ready: kubectl get csv -A | grep -E "(rhods|opendatahub|cert-manager|leader-worker)" 2. If CRs were not created (first install), run the same helm upgrade command again. 3. Check DataScienceCluster status: kubectl get datasciencecluster default-dsc -o jsonpath='{.status.phase}' === Enable Authorino TLS (Optional) === To enable TLS for Authorino: 1. Annotate the service to trigger TLS certificate generation: kubectl annotate svc/authorino-authorino-authorization \ service.beta.openshift.io/serving-cert-secret-name=authorino-server-cert \ -n kuadrant-system 2. Patch the Authorino CR to enable TLS: kubectl patch authorino/authorino -n kuadrant-system --type=merge -p '{"spec":{"clusterWide":true,"listener":{"tls":{"enabled":true,"certSecretRef":{"name":"authorino-server-cert"}}},"oidcServer":{"tls":{"enabled":false}}}}' === Documentation === For more information, see the chart README: https://github.com/opendatahub-io/odh-gitops/tree/main/charts/rhai-on-openshift-chart [INFO] Enabling bundle unpack retry on OperatorGroup cert-manager-operator/openshift-cert-manager-operator operatorgroup.operators.coreos.com/openshift-cert-manager-operator annotated [INFO] Enabling bundle unpack retry on OperatorGroup kuadrant-system/rhcl-operator operatorgroup.operators.coreos.com/rhcl-operator annotated [INFO] Enabling bundle unpack retry on OperatorGroup opendatahub-operator-system/opendatahub-operator operatorgroup.operators.coreos.com/opendatahub-operator annotated [INFO] Enabling bundle unpack retry on OperatorGroup openshift-jobset-operator/job-set operatorgroup.operators.coreos.com/job-set annotated [INFO] [INFO] Waiting for OLM to unpack operator bundles... [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator opendatahub-operator-system/opendatahub-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator opendatahub-operator-system/opendatahub-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator opendatahub-operator-system/opendatahub-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator opendatahub-operator-system/opendatahub-operator openshift-jobset-operator/job-set (2000s timeout) [INFO] Waiting for bundle unpack (4/4 Subscriptions found): cert-manager-operator/openshift-cert-manager-operator kuadrant-system/rhcl-operator (2000s timeout) [INFO] Operator bundles unpacked [INFO] [INFO] Waiting for operator CRDs to be established... ⏳ Waiting for CRD dscinitializations.dscinitialization.opendatahub.io to appear (timeout: 600s)… ✅ CRD dscinitializations.dscinitialization.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io condition met ⏳ Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 600s)… ✅ CRD datascienceclusters.datasciencecluster.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met ⏳ Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 600s)… ✅ CRD kuadrants.kuadrant.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [INFO] [INFO] Waiting for ODH operator to be ready (webhook must be serving)... [INFO] Operator CSV phase: Installing (0s / 300s) [INFO] Operator CSV is Succeeded in opendatahub-operator-system deployment.apps/opendatahub-operator-controller-manager condition met [INFO] [INFO] Applying local MaaS CRDs for kustomize mode... ⏳ Applying MaaS Controller CRDs from /workspace/source/test/maas-e2e/deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io created ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] [INFO] Phase 2: Applying CRD-dependent resources (DSC, DSCI, Kuadrant CR)... [INFO] Running: helm upgrade --install rhoai-deps Release "rhoai-deps" has been upgraded. Happy Helming! NAME: rhoai-deps LAST DEPLOYED: Mon Nov 2 18:07:23 2026 NAMESPACE: rhoai-deps STATUS: deployed REVISION: 2 TEST SUITE: None NOTES: ODH Chart installed successfully! === Installation Summary === Operator: ODH Namespace: rhoai-deps Profile: maas === Enabled Components === - aigateway: Removed - aipipelines: Removed - dashboard: Removed - feastoperator: Removed - kserve: Managed - kueue: Removed - mlflowoperator: Removed - modelregistry: Removed - ogx: Removed - ray: Removed - sparkoperator: Removed - trainer: Removed - trainingoperator: Removed - trustyai: Removed - workbenches: Removed === Enabled Services === - monitoring: Removed === Enabled Dependencies === - certManager - jobSet - rhcl === Next Steps === 1. Wait for operators to be ready: kubectl get csv -A | grep -E "(rhods|opendatahub|cert-manager|leader-worker)" 2. If CRs were not created (first install), run the same helm upgrade command again. 3. Check DataScienceCluster status: kubectl get datasciencecluster default-dsc -o jsonpath='{.status.phase}' === Enable Authorino TLS (Optional) === To enable TLS for Authorino: 1. Annotate the service to trigger TLS certificate generation: kubectl annotate svc/authorino-authorino-authorization \ service.beta.openshift.io/serving-cert-secret-name=authorino-server-cert \ -n kuadrant-system 2. Patch the Authorino CR to enable TLS: kubectl patch authorino/authorino -n kuadrant-system --type=merge -p '{"spec":{"clusterWide":true,"listener":{"tls":{"enabled":true,"certSecretRef":{"name":"authorino-server-cert"}}},"oidcServer":{"tls":{"enabled":false}}}}' === Documentation === For more information, see the chart README: https://github.com/opendatahub-io/odh-gitops/tree/main/charts/rhai-on-openshift-chart [INFO] Waiting for Kuadrant policy engine to become ready... [INFO] Waiting for: Kuadrant ready in kuadrant-system (timeout: 600s) [ERROR] Kuadrant ready in kuadrant-system - Timeout after 600s [ERROR] Kuadrant did not become ready; Authorino and policy enforcement cannot be used ❌ ERROR: MaaS platform deployment failed ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs Saved 1 resource type(s) to /workspace/artifacts-dir/maas-crs Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 5 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Skipping namespace odh-ai-gateway-infra (not found) Skipping namespace models-as-a-service (not found) Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Collecting pod logs from namespace kuadrant-system to /workspace/artifacts-dir/pod-logs/kuadrant-system Saved 6 pod log file(s) to /workspace/artifacts-dir/pod-logs/kuadrant-system Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Skipping namespace llm (not found) Skipping namespace istio-system (not found) ============================================== ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs Saved 4 resource type(s) to /workspace/artifacts-dir/maas-crs Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 5 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Skipping namespace odh-ai-gateway-infra (not found) Skipping namespace models-as-a-service (not found) Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Collecting pod logs from namespace kuadrant-system to /workspace/artifacts-dir/pod-logs/kuadrant-system Saved 6 pod log file(s) to /workspace/artifacts-dir/pod-logs/kuadrant-system Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Skipping namespace llm (not found) Skipping namespace istio-system (not found) ============================================== Collecting AIGC cluster CR extensions to /workspace/artifacts-dir/maas-crs Saved datascienceclusters.yaml Saved httproutes.yaml Saved gateways.yaml Collecting MaaS diagnostics to /workspace/artifacts-dir/gather-maas (log: /workspace/artifacts-dir/maas-must-gather.log) ... MaaS must-gather complete (see /workspace/artifacts-dir/maas-must-gather.log) ======================================== Cluster / Namespace Info ======================================== --- Current context --- admin --- Logged-in user --- system:admin --- Cluster domain --- apps.b05281def507c5a41d82.prod.konflux-ocp-ci.dev DEPLOYMENT_NAMESPACE: opendatahub MAAS_API_DEPLOYMENT_NAMESPACE: odh-ai-gateway-infra MAAS_SUBSCRIPTION_NAMESPACE: models-as-a-service AUTHORINO_NAMESPACE: kuadrant-system ======================================== MaaS API Deployment ======================================== --- maas-api pods --- --- maas-api service --- ======================================== maas-controller ======================================== --- maas-controller pods --- --- maas-controller namespace env --- namespace env=N/A ======================================== Kuadrant Policies ======================================== --- AuthPolicies (all namespaces) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE openshift-ingress openshift-ai-inference-authn False Gateway openshift-ai-inference 10m --- TokenRateLimitPolicies (all namespaces) --- ======================================== MaaS CRs (maas.opendatahub.io) ======================================== --- Configs (cluster-scoped) --- --- AITenants (all namespaces) --- --- Tenants (all namespaces) --- --- MaaSAuthPolicies --- --- MaaSSubscriptions --- --- MaaSSubscription status details --- --- MaaSModelRefs (all namespaces) --- --- MaasTenantConfigs --- --- MaasTenantConfig status details --- --- ExternalModels (maas group) --- ======================================== Inference CRs (inference.opendatahub.io) ======================================== --- ExternalModels (inference group) --- --- ExternalProviders (inference group) --- ======================================== AI Gateway CRs ======================================== --- AIGateways (cluster-scoped) --- --- LLMBatchGateways (all namespaces) --- ======================================== KServe Resources ======================================== --- LLMInferenceServices (all namespaces) --- ======================================== Kuadrant Policies (managed by maas-controller) ======================================== --- AuthPolicies (managed) --- --- TokenRateLimitPolicies (managed) --- ======================================== Payload Processing / IPP ======================================== --- IPP Deployments (openshift-ingress) --- --- IPP Services (openshift-ingress) --- --- EnvoyFilters (openshift-ingress) --- NAME AGE data-science-authn-filter 11m maas-default-gateway-authn-ssl 10m openshift-ai-inference-authn-ssl 10m --- IPP ConfigMaps (openshift-ingress) --- ======================================== NetworkPolicies ======================================== --- NetworkPolicies (openshift-ingress) --- NAME POD-SELECTOR AGE data-science-gateway-allow gateway.networking.k8s.io/gateway-name=data-science-gateway 11m istiod-allow app=istiod 12m kube-auth-proxy app=kube-auth-proxy 11m maas-default-gateway-allow gateway.networking.k8s.io/gateway-name=maas-default-gateway 12m openshift-ai-inference-allow gateway.networking.k8s.io/gateway-name=openshift-ai-inference 12m openshift-ingress-deny-all 21m router-default ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default 21m ======================================== Test User Information ======================================== No test token available (not logged in via oc) ======================================== Subscription → Model Mapping ======================================== No subscriptions found in models-as-a-service ======================================== Available Models (MaaSModelRefs) ======================================== No MaaSModelRefs found ======================================== Gateway / HTTPRoutes ======================================== --- Gateway --- NAME CLASS ADDRESS PROGRAMMED AGE maas-default-gateway openshift-default maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local True 12m --- HTTPRoutes (maas-api) --- ======================================== Authorino ======================================== --- Authorino pods --- --- ======================================== Subscription Selector Endpoint Validation ======================================== Expected URL (from resolved maas-api deployment namespace): https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select (MAAS_API_DEPLOYMENT_NAMESPACE resolved to: odh-ai-gateway-infra) --- Sample AuthPolicy subscription-info configuration --- No managed AuthPolicies found --- Connectivity test (from kuadrant-system, simulates Authorino) --- curl -vsk -m 10 -X POST 'https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select' -H 'Content-Type: application/json' -d '{}' kubectl run failed or timed out ======================================== DNS Resolution Check ======================================== Resolving: maas-api.odh-ai-gateway-infra.svc.cluster.local nslookup failed ======================================== Configuration Summary ======================================== This summary helps compare local vs CI runs: AITenants (all namespaces): 0 MaaSModelRefs (all namespaces): 0 MaaSSubscriptions (models-as-a-service): 0 MaaSAuthPolicies (models-as-a-service): 0 Generated Kuadrant AuthPolicies: 0 Generated Kuadrant TokenRateLimitPolicies: 0 LLMInferenceServices (all namespaces): 0 ExternalModels - inference (all namespaces): 0 ExternalProviders - inference (all namespaces): 0 EnvoyFilters (openshift-ingress): 3 IPP Deployments (openshift-ingress): 0 NetworkPolicies (openshift-ingress): 7 Subscription selector URL: https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select Test user: system:admin