Fetching https://github.com/jland-redhat/models-as-a-service@630e7b50303688a6e3f25cd039e32822f38d6e28 into /workspace/source/test/maas-e2e ... MaaS checkout at: 630e7b50303688a6e3f25cd039e32822f38d6e28 Patched /workspace/source/test/maas-e2e/scripts/deploy.sh Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS AuthPolicy wait scope) MaaS platform images: MAAS_API_IMAGE=quay.io/opendatahub/maas-api:latest MAAS_CONTROLLER_IMAGE=quay.io/opendatahub/maas-controller:latest AI_GATEWAY_CONTROLLER_IMAGE=quay.io/opendatahub/odh-ai-gateway-controller@sha256:14f47984e2de64cfd0b6cb4892c02cfa94fd41461dce12c7da0244fc33685f0b PRAXIS_EXTPROC_IMAGE=quay.io/maas/odh-praxis-extproc:pr699-76cb977 (praxis-proxy/ai#699 @ 76cb977ad65235a9d965efc014c87bcad00e4960) ---------------------------------------- ai-gateway-controller E2E on OpenShift ---------------------------------------- Checking prerequisites... Prerequisites met — logged in as: system:admin DEPLOY_MODE: kustomize MAAS_API_IMAGE: quay.io/opendatahub/maas-api:latest MAAS_CONTROLLER_IMAGE: quay.io/opendatahub/maas-controller:latest AI_GATEWAY_CONTROLLER_IMAGE: quay.io/opendatahub/odh-ai-gateway-controller@sha256:14f47984e2de64cfd0b6cb4892c02cfa94fd41461dce12c7da0244fc33685f0b PRAXIS_EXTPROC_IMAGE: quay.io/maas/odh-praxis-extproc:pr699-76cb977 2026-09-15T17:35:04Z deploy_platform start Deploying MaaS platform via ODH operator... Gateway ingress mode for deploy.sh: ocproute (loadbalancer | ocproute) Using custom MaaS API image: quay.io/opendatahub/maas-api:latest Using custom MaaS controller image: quay.io/opendatahub/maas-controller:latest Deployment mode: kustomize Installing cert-manager and LeaderWorkerSet operators... === Installing cert-manager and LeaderWorkerSet operators === 1. Installing cert-manager operator... namespace/cert-manager-operator created operatorgroup.operators.coreos.com/cert-manager-operator created subscription.operators.coreos.com/openshift-cert-manager-operator created Waiting for Subscription cert-manager-operator/openshift-cert-manager-operator... subscription.operators.coreos.com/openshift-cert-manager-operator condition met Waiting for CSV cert-manager-operator.v1.20.0 to succeed... clusterserviceversion.operators.coreos.com/cert-manager-operator.v1.20.0 condition met cert-manager ready. 2. Installing LeaderWorkerSet operator... namespace/openshift-lws-operator created operatorgroup.operators.coreos.com/leader-worker-set created subscription.operators.coreos.com/leader-worker-set created Waiting for Subscription openshift-lws-operator/leader-worker-set... subscription.operators.coreos.com/leader-worker-set condition met Waiting for CSV leader-worker-set.v1.0.0 to succeed... clusterserviceversion.operators.coreos.com/leader-worker-set.v1.0.0 condition met LeaderWorkerSet operator ready. 3. Activating LeaderWorkerSet API... leaderworkersetoperator.operator.openshift.io/cluster created LeaderWorkerSetOperator CR applied. === Done === Verify: oc get pods -n cert-manager-operator oc get pods -n openshift-lws-operator oc get crd leaderworkersets.leaderworkerset.x-k8s.io Installing OpenDataHub operator... === Installing OpenDataHub operator === 1. Setting up ODH catalog... Using community-operators 2. Installing ODH operator... [INFO] Installing operator: opendatahub-operator in namespace: opendatahub [INFO] Creating namespace: opendatahub namespace/opendatahub created namespace/opendatahub condition met [INFO] Creating OperatorGroup in opendatahub for AllNamespaces mode operatorgroup.operators.coreos.com/opendatahub-operatorgroup created [INFO] Creating Subscription for opendatahub-operator from community-operators (channel: fast-3, installPlanApproval: Manual, startingCSV: opendatahub-operator.v3.5.0-ea.2) subscription.operators.coreos.com/opendatahub-operator created [INFO] Manual Subscription: approving initial InstallPlan so first install can proceed... [INFO] Approving initial InstallPlan install-lxqmh (Manual subscription) installplan.operators.coreos.com/install-lxqmh patched [INFO] Waiting for subscription to install... * Waiting for Subscription opendatahub/opendatahub-operator to start setup... subscription.operators.coreos.com/opendatahub-operator condition met * Waiting for Subscription setup to finish setup. CSV = opendatahub-operator.v3.5.0-ea.2 ... clusterserviceversion.operators.coreos.com/opendatahub-operator.v3.5.0-ea.2 condition met [INFO] Operator opendatahub-operator installed successfully 3. Skipping operator image patch (OPERATOR_IMAGE not set) 4. Waiting for operator CRDs... ⏳ Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 180s)… ✅ CRD datascienceclusters.datasciencecluster.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met 5. Waiting for operator webhook... * Waiting for deployment/opendatahub-operator-controller-manager in opendatahub (timeout: 120s)... * Found deployment/opendatahub-operator-controller-manager deployment.apps/opendatahub-operator-controller-manager condition met 6. Applying DSCInitialization... dscinitialization.dscinitialization.opendatahub.io/default-dsci created Waiting for DSCInitialization to be Ready... Waiting for DSCInitialization Ready (attempt 1/30, phase=Progressing, Ready=unknown)... DSCInitialization is Ready 7. Applying DataScienceCluster... datasciencecluster.datasciencecluster.opendatahub.io/default-dsc serverside-applied 8. Waiting for DataScienceCluster (KServe)... * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... - KServe state: , KserveReady: , ModelsAsServiceReady: (informational only), AIGatewayReady: - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: False (informational only), AIGatewayReady: - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: False (informational only), AIGatewayReady: * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' 9. Waiting for odh-model-controller webhook... ⏳ Waiting for validating webhooks in namespace opendatahub (timeout: 180s)... ✅ Webhook service opendatahub/kserve-webhook-server-service has ready endpoints ✅ Webhook service opendatahub/llmisvc-webhook-server-service has ready endpoints ✅ Webhook service opendatahub/odh-model-controller-webhook-service has ready endpoints ✅ Webhook service opendatahub/opendatahub-operator-controller-manager-service has ready endpoints 🎉 All validating webhook services in opendatahub are ready === ODH installation complete === Verify: kubectl get datasciencecluster -A kubectl get pods -n opendatahub kubectl get pods -n kserve Using policy engine: rhcl (Authorino namespace: kuadrant-system) [INFO] =================================================== [INFO] Models-as-a-Service Deployment [INFO] =================================================== [INFO] Validating configuration... [INFO] Configuration validated successfully [INFO] Deployment configuration: [INFO] Mode: kustomize [INFO] Policy Engine: rhcl [INFO] Namespace: opendatahub [INFO] TLS Backend: true [INFO] External OIDC: false [INFO] MaaS API image: quay.io/opendatahub/maas-api:latest [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:latest [INFO] Starting kustomize-based deployment... [INFO] Installing policy engine: rhcl [INFO] Installing RHCL (Red Hat Connectivity Link - downstream) [INFO] Using RHCL channel head from redhat-operators (stable) [INFO] Installing RHCL into namespace: kuadrant-system [INFO] Installing operator: rhcl-operator in namespace: kuadrant-system [INFO] Creating namespace: kuadrant-system namespace/kuadrant-system created namespace/kuadrant-system condition met [INFO] Creating OperatorGroup in kuadrant-system for AllNamespaces mode operatorgroup.operators.coreos.com/kuadrant-system-operatorgroup created [INFO] Creating Subscription for rhcl-operator from redhat-operators (channel: stable) subscription.operators.coreos.com/rhcl-operator created [INFO] Waiting for subscription to install... * Waiting for Subscription kuadrant-system/rhcl-operator to start setup... subscription.operators.coreos.com/rhcl-operator condition met * Waiting for Subscription setup to finish setup. CSV = rhcl-operator.v1.4.3 ... clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 condition met [INFO] Operator rhcl-operator installed successfully [INFO] Patching rhcl-operator CSV (Gateway API, rate limit failure modes, auth service timeout)... clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched [INFO] CSV patched (Gateway controller and/or rate limit failure modes and/or auth timeout) [INFO] Forcing operator restart to apply CSV env configuration... pod "kuadrant-operator-controller-manager-76f8f686bf-wwwm4" force deleted pod "kuadrant-operator-controller-manager-7978cbdfbc-ghgh2" force deleted pod "limitador-operator-controller-manager-74495c69f8-z82kh" force deleted [INFO] Waiting for operator pod to restart... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... deployment "kuadrant-operator-controller-manager" successfully rolled out [WARN] Operator pod may not have correct env yet (ISTIO / RATELIMIT_* failure modes / AUTH_SERVICE_TIMEOUT) [INFO] Waiting 15s for operator to fully initialize with Gateway controller configuration... [INFO] Initializing Gateway API and ModelsAsAService gateway... [INFO] =================================================== [INFO] MaaS Gateway Setup [INFO] =================================================== [INFO] Validating gateway configuration... [INFO] Configuration validated [INFO] Ingress mode: ocproute [INFO] Disconnected: false [INFO] AllowedRoutes: namespaces=opendatahub,odh-ai-gateway-infra,llm [INFO] Detecting cluster domain... [INFO] Detected cluster domain: apps.f06f72d5-aad5-4cbc-baba-955862daaaaf.prod.konfluxeaas.com [INFO] Setting up GatewayClass... [INFO] Creating GatewayClass openshift-default... gatewayclass.gateway.networking.k8s.io/openshift-default created [INFO] Setting up Gateway in ocproute mode (ClusterIP with OpenShift Route)... [INFO] Creating ConfigMap gw-options... configmap/gw-options created [INFO] Creating/updating Gateway maas-default-gateway (ocproute mode)... gateway.gateway.networking.k8s.io/maas-default-gateway serverside-applied [INFO] Waiting for Gateway to be Programmed (timeout: 120s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met [INFO] Gateway is Programmed [INFO] Gateway Service ready: maas-default-gateway-openshift-default [INFO] Creating Route maas-gateway-route... [INFO] Host: maas.apps.f06f72d5-aad5-4cbc-baba-955862daaaaf.prod.konfluxeaas.com [INFO] Target Service: maas-default-gateway-openshift-default route.route.openshift.io/maas-gateway-route created [INFO] Waiting for Route to be Admitted... [INFO] Route is Admitted [INFO] ClusterIP mode setup complete [INFO] [INFO] =================================================== [INFO] Gateway setup completed successfully [INFO] =================================================== [INFO] Mode: ocproute [INFO] Gateway: maas-default-gateway [INFO] Namespace: openshift-ingress [INFO] Route: maas-gateway-route [INFO] Hostname: maas.apps.f06f72d5-aad5-4cbc-baba-955862daaaaf.prod.konfluxeaas.com [INFO] [INFO] Verify with: [INFO] kubectl get gateway maas-default-gateway -n openshift-ingress [INFO] kubectl get route maas-gateway-route -n openshift-ingress [INFO] Applying Kuadrant custom resource in kuadrant-system... kuadrant.kuadrant.io/kuadrant created [INFO] Waiting for Kuadrant to become ready (initial check)... [INFO] Waiting for: Kuadrant ready in kuadrant-system (timeout: 60s) [INFO] Kuadrant ready in kuadrant-system - Ready [INFO] Kuadrant setup complete [INFO] Ensuring namespace exists: opendatahub [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [WARN] DEPLOYING POC POSTGRESQL — NOT INTENDED FOR PRODUCTION USE [WARN] Data is stored in ephemeral storage and will be lost on pod restart. [WARN] For production, use --postgres-connection with an external database [WARN] (AWS RDS, Crunchy Operator, Azure Database, etc.) [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ⚠️ WARNING FOR PRODUCTION USE. ⚠️ ┃ ┃ This deploys PostgreSQL with ephemeral storage (emptyDir). ┃ ┃ Data WILL be lost on pod restart. ┃ ┃ For production, use an external database: ┃ ┃ deploy.sh --postgres-connection postgresql://... ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 🔧 Fresh install: Deploying PostgreSQL in infrastructure namespace 'odh-ai-gateway-infra'... 📦 Creating infrastructure namespace 'odh-ai-gateway-infra'... namespace/odh-ai-gateway-infra created Generated random PostgreSQL password (stored in secret postgres-creds) Creating PostgreSQL deployment... ⚠️ Using POC configuration (ephemeral storage) Using default PostgreSQL image (operator CSV not available) Image: registry.redhat.io/rhel9/postgresql-16:latest secret/postgres-creds created deployment.apps/postgres created service/postgres created secret/maas-db-config created Waiting for PostgreSQL to be ready... deployment.apps/postgres condition met ✅ PostgreSQL deployed successfully Namespace: odh-ai-gateway-infra Database: maas User: maas Secret: maas-db-config (contains DB_CONNECTION_URL with FQDN) ⚠️ For production, use AWS RDS, Crunchy Operator, or Azure Database Note: Schema migrations run automatically when maas-api starts [INFO] Configuring TLS backend for Authorino and MaaS API... * Waiting for deployment/authorino in kuadrant-system (timeout: 300s)... * Found deployment/authorino [INFO] Running TLS configuration script... [INFO] TLS configuration script completed successfully [INFO] Restarting deployments to pick up TLS configuration... deployment.apps/authorino restarted [INFO] Waiting for Authorino deployment to be ready... Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out [INFO] TLS backend configuration complete [INFO] Kustomize prerequisite deployment completed [INFO] [INFO] MaaS Controller... [INFO] Phase 1: Applying MaaS CRDs and waiting until Established (controller creates Config after CRD is ready)... ⏳ Applying MaaS Controller CRDs from /workspace/source/test/maas-e2e/scripts/../deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io created ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] Phase 2: Applying full controller kustomize (same as operator: deployment/base/maas-controller/default)... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io configured serviceaccount/maas-controller created role.rbac.authorization.k8s.io/maas-controller-leader-election-role created clusterrole.rbac.authorization.k8s.io/maas-controller-cluster-config-role created clusterrole.rbac.authorization.k8s.io/maas-controller-ocp-role created clusterrole.rbac.authorization.k8s.io/maas-controller-role created clusterrole.rbac.authorization.k8s.io/maas-owner-role created clusterrole.rbac.authorization.k8s.io/maas-viewer-role created rolebinding.rbac.authorization.k8s.io/maas-controller-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-cluster-config-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-ocp-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-rolebinding created configmap/maas-parameters created service/maas-controller-metrics created service/maas-controller-webhook-service created deployment.apps/maas-controller created servicemonitor.monitoring.coreos.com/maas-controller-metrics created networkpolicy.networking.k8s.io/maas-controller-allow-monitoring created validatingwebhookconfiguration.admissionregistration.k8s.io/maas-validating-webhook-configuration created [INFO] Restarting maas-controller to pick up manifest and ConfigMap changes deployment.apps/maas-controller restarted [INFO] Waiting for maas-controller to be ready... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... deployment "maas-controller" successfully rolled out [INFO] Controller ready. [INFO] [INFO] Waiting for Tenant reconciler to deploy maas-api... [INFO] Applying secret migration RBAC to namespace odh-ai-gateway-infra... role.rbac.authorization.k8s.io/maas-controller-secret-migrate created rolebinding.rbac.authorization.k8s.io/maas-controller-secret-migrate created [INFO] maas-api deployment found in odh-ai-gateway-infra, waiting for rollout... Waiting for deployment "maas-api" rollout to finish: 0 of 1 updated replicas are available... deployment "maas-api" successfully rolled out [INFO] maas-api is ready [INFO] [INFO] MaaS API and MaaS Controller deployment completed successfully! [INFO] maas-api image: quay.io/opendatahub/maas-api:latest (namespace: odh-ai-gateway-infra) [INFO] maas-controller image: quay.io/opendatahub/maas-controller:latest (namespace: opendatahub) [INFO] =================================================== [INFO] Models-as-a-Service Deployment completed successfully! [INFO] =================================================== * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' ⚠️ WARNING: Skipping Authorino readiness check (SKIP_AUTH_CHECK=true) ✅ MaaS platform deployment completed 2026-09-15T17:40:31Z deploy_platform end ---------------------------------------- Deploying Models ---------------------------------------- 2026-09-15T17:40:31Z deploy_models start Deploying MaaS system (free + premium: LLMIS + MaaSModelRef + MaaSAuthPolicy + MaaSSubscription) Waiting for Gateway openshift-ingress/maas-default-gateway to be Programmed=True (timeout: 600s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met ✅ Gateway openshift-ingress/maas-default-gateway is Programmed Creating 'llm' namespace... namespace/llm created 'models-as-a-service' namespace already exists Error: must build at directory: not a valid directory: evalsymlink failure on 'test/e2e/fixtures/' : lstat /workspace/source/test/e2e/fixtures: no such file or directory error: no objects passed to apply ❌ ERROR: Failed to deploy MaaS system with e2e fixtures ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Saved to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs Saved 3 resource type(s) to /workspace/artifacts-dir/maas-crs Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 8 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Collecting pod logs from namespace odh-ai-gateway-infra to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Saved 2 pod log file(s) to /workspace/artifacts-dir/pod-logs/odh-ai-gateway-infra Collecting pod logs from namespace models-as-a-service to /workspace/artifacts-dir/pod-logs/models-as-a-service Saved 0 0 pod log file(s) to /workspace/artifacts-dir/pod-logs/models-as-a-service Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Collecting pod logs from namespace kuadrant-system to /workspace/artifacts-dir/pod-logs/kuadrant-system Saved 7 pod log file(s) to /workspace/artifacts-dir/pod-logs/kuadrant-system Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 8 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Collecting pod logs from namespace llm to /workspace/artifacts-dir/pod-logs/llm Saved 0 0 pod log file(s) to /workspace/artifacts-dir/pod-logs/llm Skipping namespace istio-system (not found) ============================================== ======================================== Cluster / Namespace Info ======================================== --- Current context --- admin --- Logged-in user --- system:admin --- Cluster domain --- apps.f06f72d5-aad5-4cbc-baba-955862daaaaf.prod.konfluxeaas.com DEPLOYMENT_NAMESPACE: opendatahub MAAS_API_DEPLOYMENT_NAMESPACE: odh-ai-gateway-infra MAAS_SUBSCRIPTION_NAMESPACE: models-as-a-service AUTHORINO_NAMESPACE: kuadrant-system ======================================== MaaS API Deployment ======================================== --- maas-api pods --- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES maas-api-59f6566bbc-xs8mq 1/1 Running 0 22s 10.133.0.48 ip-10-0-141-84.ec2.internal --- maas-api service --- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR maas-api ClusterIP 172.31.132.239 8443/TCP 22s app.kubernetes.io/component=api,app.kubernetes.io/name=maas-api,app.kubernetes.io/part-of=models-as-a-service,maas.opendatahub.io/tenant-instance=maas-api ======================================== maas-controller ======================================== --- maas-controller pods --- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES maas-controller-5dbbcdb4c-kjft7 1/1 Running 0 59s 10.133.0.47 ip-10-0-141-84.ec2.internal --- maas-controller namespace env --- INFRA_NAMESPACE=AUTO ======================================== Kuadrant Policies ======================================== --- AuthPolicies (all namespaces) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE openshift-ingress maas-gateway-auth True True Gateway maas-default-gateway 53s --- TokenRateLimitPolicies (all namespaces) --- NAMESPACE NAME ACCEPTED ENFORCED TARGETKIND TARGETNAME TARGETSECTION AGE openshift-ingress gateway-default-deny True True Gateway maas-default-gateway 22s ======================================== MaaS CRs (maas.opendatahub.io) ======================================== --- Configs (cluster-scoped) --- NAME READY AGE default True 54s --- AITenants (all namespaces) --- NAMESPACE NAME READY TENANT NAMESPACE GATEWAY AGE ai-tenants models-as-a-service True models-as-a-service maas-default-gateway 25s --- Tenants (all namespaces) --- --- MaaSAuthPolicies --- --- MaaSSubscriptions --- --- MaaSSubscription status details --- --- MaaSModelRefs (all namespaces) --- --- MaasTenantConfigs --- NAME READY REASON AGE INFRANAMESPACE default-tenant True Reconciled 24s odh-ai-gateway-infra --- MaasTenantConfig status details --- default-tenant: True - MaaS platform manifests applied and maas-api deployment is available --- ExternalModels (maas group) --- ======================================== Inference CRs (inference.opendatahub.io) ======================================== --- ExternalModels (inference group) --- --- ExternalProviders (inference group) --- ======================================== AI Gateway CRs ======================================== --- AIGateways (cluster-scoped) --- --- LLMBatchGateways (all namespaces) --- ======================================== Test User Information ======================================== No test token available (not logged in via oc) ======================================== Subscription → Model Mapping ======================================== No subscriptions found in models-as-a-service ======================================== Available Models (MaaSModelRefs) ======================================== No MaaSModelRefs found ======================================== Gateway / HTTPRoutes ======================================== --- Gateway --- NAME CLASS ADDRESS PROGRAMMED AGE maas-default-gateway openshift-default maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local True 115s --- HTTPRoutes (maas-api) --- NAME HOSTNAMES AGE maas-api-route 24s ======================================== Authorino ======================================== --- Authorino pods --- --- ======================================== Subscription Selector Endpoint Validation ======================================== Expected URL (from resolved maas-api deployment namespace): https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select (MAAS_API_DEPLOYMENT_NAMESPACE resolved to: odh-ai-gateway-infra) --- Sample AuthPolicy subscription-info configuration --- Inspecting: openshift-ingress/maas-gateway-auth Actual URL in AuthPolicy: https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select ✅ Request body includes requestedModel field Model reference: / Request body preview: { "groups": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.groups : (has(auth.identity.groups) ? auth.identity.groups : auth.identity.user.groups), "username": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.username : (has(auth.identity.preferred_username) ? auth.identity.preferred_username : (has(auth.identity.sub) ? auth.identity.sub : auth.identity.user.username)), "requestedSubscription": (has(auth.metadata) && has(auth.metadata.apiKeyValidation)) ? auth.metadata.apiKeyValidation.subscription : ("x-maas-subscription" in request.headers ? request.headers["x-maas-subscription"] : ""), "requestedModel": (size(request.path.split("/").filter(x, x != "")) >= 2 && request.path.split("/").filter(x, x != "")[0] != "v1" && request.path.split("/").filter(x, x != "")[0] != "maas-api" ? request.path.split("/").filter(x, x != "")[0] + "/" + request.path.split("/").filter(x, x != "")[1] : ("x-gateway-model-name" in request.headers ? request.headers["x-gateway-model-name"] : "")) --- Connectivity test (from kuadrant-system, simulates Authorino) --- curl -vsk -m 10 -X POST 'https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select' -H 'Content-Type: application/json' -d '{}' kubectl run failed or timed out ======================================== DNS Resolution Check ======================================== Resolving: maas-api.odh-ai-gateway-infra.svc.cluster.local nslookup failed ======================================== Configuration Summary ======================================== This summary helps compare local vs CI runs: AITenants (all namespaces): 1 MaaSModelRefs (all namespaces): 0 MaaSSubscriptions (models-as-a-service): 0 MaaSAuthPolicies (models-as-a-service): 0 Generated Kuadrant AuthPolicies: 1 ExternalModels - inference (all namespaces): 0 ExternalProviders - inference (all namespaces): 0 Subscription selector URL: https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select Test user: system:admin