Fetching https://github.com/opendatahub-io/models-as-a-service@5c36d49a5c1bcd668e9f5bb995a7599bceb6125a into /workspace/source/test/maas-e2e ... MaaS checkout at: 5c36d49a5c1bcd668e9f5bb995a7599bceb6125a Patched /workspace/source/test/maas-e2e/scripts/deploy.sh Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS checkout fixture root) Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS AuthPolicy wait scope) MaaS platform images: MAAS_API_IMAGE=quay.io/opendatahub/maas-api:latest MAAS_CONTROLLER_IMAGE=quay.io/opendatahub/maas-controller:latest AI_GATEWAY_CONTROLLER_IMAGE=quay.io/opendatahub/odh-ai-gateway-controller@sha256:759d42a8e41c93de070f7adc81bbc3ffeac823c7b48fa6227e005c8848a64ffc PRAXIS_EXTPROC_IMAGE=quay.io/opendatahub/odh-praxis-extproc:odh-stable ---------------------------------------- ai-gateway-controller E2E on OpenShift ---------------------------------------- Checking prerequisites... Prerequisites met — logged in as: system:admin DEPLOY_MODE: kustomize MAAS_API_IMAGE: quay.io/opendatahub/maas-api:latest MAAS_CONTROLLER_IMAGE: quay.io/opendatahub/maas-controller:latest AI_GATEWAY_CONTROLLER_IMAGE: quay.io/opendatahub/odh-ai-gateway-controller@sha256:759d42a8e41c93de070f7adc81bbc3ffeac823c7b48fa6227e005c8848a64ffc PRAXIS_EXTPROC_IMAGE: quay.io/opendatahub/odh-praxis-extproc:odh-stable 2026-09-18T20:17:02Z deploy_platform start Deploying MaaS platform via ODH operator... Gateway ingress mode for deploy.sh: ocproute (loadbalancer | ocproute) Using custom MaaS API image: quay.io/opendatahub/maas-api:latest Using custom MaaS controller image: quay.io/opendatahub/maas-controller:latest Deployment mode: kustomize Installing cert-manager and LeaderWorkerSet operators... === Installing cert-manager and LeaderWorkerSet operators === 1. Installing cert-manager operator... namespace/cert-manager-operator created operatorgroup.operators.coreos.com/cert-manager-operator created subscription.operators.coreos.com/openshift-cert-manager-operator created Waiting for Subscription cert-manager-operator/openshift-cert-manager-operator... subscription.operators.coreos.com/openshift-cert-manager-operator condition met Waiting for CSV cert-manager-operator.v1.20.0 to succeed... clusterserviceversion.operators.coreos.com/cert-manager-operator.v1.20.0 condition met cert-manager ready. 2. Installing LeaderWorkerSet operator... namespace/openshift-lws-operator created operatorgroup.operators.coreos.com/leader-worker-set created subscription.operators.coreos.com/leader-worker-set created Waiting for Subscription openshift-lws-operator/leader-worker-set... subscription.operators.coreos.com/leader-worker-set condition met Waiting for CSV leader-worker-set.v1.0.0 to succeed... clusterserviceversion.operators.coreos.com/leader-worker-set.v1.0.0 condition met LeaderWorkerSet operator ready. 3. Activating LeaderWorkerSet API... leaderworkersetoperator.operator.openshift.io/cluster created LeaderWorkerSetOperator CR applied. === Done === Verify: oc get pods -n cert-manager-operator oc get pods -n openshift-lws-operator oc get crd leaderworkersets.leaderworkerset.x-k8s.io Installing OpenDataHub operator... === Installing OpenDataHub operator === 1. Setting up ODH catalog... Using community-operators 2. Installing ODH operator... [INFO] Installing operator: opendatahub-operator in namespace: opendatahub [INFO] Creating namespace: opendatahub namespace/opendatahub created namespace/opendatahub condition met [INFO] Creating OperatorGroup in opendatahub for AllNamespaces mode operatorgroup.operators.coreos.com/opendatahub-operatorgroup created [INFO] Creating Subscription for opendatahub-operator from community-operators (channel: fast-3, installPlanApproval: Manual, startingCSV: opendatahub-operator.v3.5.0-ea.2) subscription.operators.coreos.com/opendatahub-operator created [INFO] Manual Subscription: approving initial InstallPlan so first install can proceed... [INFO] Approving initial InstallPlan install-shqb5 (Manual subscription) installplan.operators.coreos.com/install-shqb5 patched [INFO] Waiting for subscription to install... * Waiting for Subscription opendatahub/opendatahub-operator to start setup... subscription.operators.coreos.com/opendatahub-operator condition met * Waiting for Subscription setup to finish setup. CSV = opendatahub-operator.v3.5.0-ea.2 ... clusterserviceversion.operators.coreos.com/opendatahub-operator.v3.5.0-ea.2 condition met [INFO] Operator opendatahub-operator installed successfully 3. Skipping operator image patch (OPERATOR_IMAGE not set) 4. Waiting for operator CRDs... ⏳ Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 180s)… ✅ CRD datascienceclusters.datasciencecluster.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met 5. Waiting for operator webhook... * Waiting for deployment/opendatahub-operator-controller-manager in opendatahub (timeout: 120s)... * Found deployment/opendatahub-operator-controller-manager deployment.apps/opendatahub-operator-controller-manager condition met 6. Applying DSCInitialization... dscinitialization.dscinitialization.opendatahub.io/default-dsci created Waiting for DSCInitialization to be Ready... Waiting for DSCInitialization Ready (attempt 1/30, phase=unknown, Ready=unknown)... DSCInitialization is Ready 7. Applying DataScienceCluster... datasciencecluster.datasciencecluster.opendatahub.io/default-dsc serverside-applied 8. Waiting for DataScienceCluster (KServe)... * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... - KServe state: , KserveReady: , ModelsAsServiceReady: (informational only), AIGatewayReady: - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: False (informational only), AIGatewayReady: - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: False (informational only), AIGatewayReady: * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' 9. Waiting for odh-model-controller webhook... ⏳ Waiting for validating webhooks in namespace opendatahub (timeout: 180s)... ✅ Webhook service opendatahub/kserve-webhook-server-service has ready endpoints ✅ Webhook service opendatahub/llmisvc-webhook-server-service has ready endpoints ✅ Webhook service opendatahub/odh-model-controller-webhook-service has ready endpoints ✅ Webhook service opendatahub/opendatahub-operator-controller-manager-service has ready endpoints 🎉 All validating webhook services in opendatahub are ready === ODH installation complete === Verify: kubectl get datasciencecluster -A kubectl get pods -n opendatahub kubectl get pods -n kserve Using policy engine: rhcl (Authorino namespace: kuadrant-system) [INFO] =================================================== [INFO] Models-as-a-Service Deployment [INFO] =================================================== [INFO] Validating configuration... [INFO] Configuration validated successfully [INFO] Deployment configuration: [INFO] Mode: kustomize [INFO] Policy Engine: rhcl [INFO] Namespace: opendatahub [INFO] TLS Backend: true [INFO] External OIDC: false [INFO] MaaS API image: quay.io/opendatahub/maas-api:latest [INFO] MaaS controller image: quay.io/opendatahub/maas-controller:latest [INFO] Starting kustomize-based deployment... [INFO] Installing policy engine: rhcl [INFO] Installing RHCL (Red Hat Connectivity Link - downstream) [INFO] Using RHCL channel head from redhat-operators (stable) [INFO] Installing RHCL into namespace: kuadrant-system [INFO] Installing operator: rhcl-operator in namespace: kuadrant-system [INFO] Creating namespace: kuadrant-system namespace/kuadrant-system created namespace/kuadrant-system condition met [INFO] Creating OperatorGroup in kuadrant-system for AllNamespaces mode operatorgroup.operators.coreos.com/kuadrant-system-operatorgroup created [INFO] Creating Subscription for rhcl-operator from redhat-operators (channel: stable) subscription.operators.coreos.com/rhcl-operator created [INFO] Waiting for subscription to install... * Waiting for Subscription kuadrant-system/rhcl-operator to start setup... subscription.operators.coreos.com/rhcl-operator condition met * Waiting for Subscription setup to finish setup. CSV = rhcl-operator.v1.4.3 ... clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 condition met [INFO] Operator rhcl-operator installed successfully [INFO] Patching rhcl-operator CSV (Gateway API, rate limit failure modes, auth service timeout)... clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched clusterserviceversion.operators.coreos.com/rhcl-operator.v1.4.3 patched [INFO] CSV patched (Gateway controller and/or rate limit failure modes and/or auth timeout) [INFO] Forcing operator restart to apply CSV env configuration... pod "kuadrant-operator-controller-manager-76f8f686bf-jf2bb" force deleted pod "limitador-operator-controller-manager-74495c69f8-rctz9" force deleted [INFO] Waiting for operator pod to restart... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination... deployment "kuadrant-operator-controller-manager" successfully rolled out [INFO] Operator pod has required CSV env (ISTIO gateway controller + RATELIMIT_* failure modes + AUTH_SERVICE_TIMEOUT) [INFO] Waiting 15s for operator to fully initialize with Gateway controller configuration... [INFO] Initializing Gateway API and ModelsAsAService gateway... [INFO] =================================================== [INFO] MaaS Gateway Setup [INFO] =================================================== [INFO] Validating gateway configuration... [INFO] Configuration validated [INFO] Ingress mode: ocproute [INFO] Disconnected: false [INFO] AllowedRoutes: namespaces=opendatahub,odh-ai-gateway-infra,llm [INFO] Detecting cluster domain... [INFO] Detected cluster domain: apps.fff2ee93-189b-4388-ad84-9da3d9b5d7a1.prod.konfluxeaas.com [INFO] Setting up GatewayClass... [INFO] Creating GatewayClass openshift-default... gatewayclass.gateway.networking.k8s.io/openshift-default created [INFO] Setting up Gateway in ocproute mode (ClusterIP with OpenShift Route)... [INFO] Creating ConfigMap gw-options... configmap/gw-options created [INFO] Creating/updating Gateway maas-default-gateway (ocproute mode)... gateway.gateway.networking.k8s.io/maas-default-gateway serverside-applied [INFO] Waiting for Gateway to be Programmed (timeout: 120s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met [INFO] Gateway is Programmed [INFO] Gateway Service ready: maas-default-gateway-openshift-default [INFO] Creating Route maas-gateway-route... [INFO] Host: maas.apps.fff2ee93-189b-4388-ad84-9da3d9b5d7a1.prod.konfluxeaas.com [INFO] Target Service: maas-default-gateway-openshift-default route.route.openshift.io/maas-gateway-route created [INFO] Waiting for Route to be Admitted... [INFO] Route is Admitted [INFO] ClusterIP mode setup complete [INFO] [INFO] =================================================== [INFO] Gateway setup completed successfully [INFO] =================================================== [INFO] Mode: ocproute [INFO] Gateway: maas-default-gateway [INFO] Namespace: openshift-ingress [INFO] Route: maas-gateway-route [INFO] Hostname: maas.apps.fff2ee93-189b-4388-ad84-9da3d9b5d7a1.prod.konfluxeaas.com [INFO] [INFO] Verify with: [INFO] kubectl get gateway maas-default-gateway -n openshift-ingress [INFO] kubectl get route maas-gateway-route -n openshift-ingress [INFO] Applying Kuadrant custom resource in kuadrant-system... kuadrant.kuadrant.io/kuadrant created [INFO] Waiting for Kuadrant to become ready (initial check)... [INFO] Waiting for: Kuadrant ready in kuadrant-system (timeout: 60s) [INFO] Kuadrant ready in kuadrant-system - Ready [INFO] Kuadrant setup complete [INFO] Ensuring namespace exists: opendatahub [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [WARN] DEPLOYING POC POSTGRESQL — NOT INTENDED FOR PRODUCTION USE [WARN] Data is stored in ephemeral storage and will be lost on pod restart. [WARN] For production, use --postgres-connection with an external database [WARN] (AWS RDS, Crunchy Operator, Azure Database, etc.) [WARN] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ⚠️ WARNING FOR PRODUCTION USE. ⚠️ ┃ ┃ This deploys PostgreSQL with ephemeral storage (emptyDir). ┃ ┃ Data WILL be lost on pod restart. ┃ ┃ For production, use an external database: ┃ ┃ deploy.sh --postgres-connection postgresql://... ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ 🔧 Fresh install: Deploying PostgreSQL in infrastructure namespace 'odh-ai-gateway-infra'... 📦 Creating infrastructure namespace 'odh-ai-gateway-infra'... namespace/odh-ai-gateway-infra created Generated random PostgreSQL password (stored in secret postgres-creds) Creating PostgreSQL deployment... ⚠️ Using POC configuration (ephemeral storage) Using default PostgreSQL image (operator CSV not available) Image: registry.redhat.io/rhel9/postgresql-16:latest secret/postgres-creds created deployment.apps/postgres created service/postgres created secret/maas-db-config created Waiting for PostgreSQL to be ready... deployment.apps/postgres condition met ✅ PostgreSQL deployed successfully Namespace: odh-ai-gateway-infra Database: maas User: maas Secret: maas-db-config (contains DB_CONNECTION_URL with FQDN) ⚠️ For production, use AWS RDS, Crunchy Operator, or Azure Database Note: Schema migrations run automatically when maas-api starts [INFO] Configuring TLS backend for Authorino and MaaS API... * Waiting for deployment/authorino in kuadrant-system (timeout: 300s)... * Found deployment/authorino [INFO] Running TLS configuration script... [INFO] TLS configuration script completed successfully [INFO] Restarting deployments to pick up TLS configuration... deployment.apps/authorino restarted [INFO] Waiting for Authorino deployment to be ready... Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out [INFO] TLS backend configuration complete [INFO] Kustomize prerequisite deployment completed [INFO] [INFO] MaaS Controller... [INFO] Phase 1: Applying MaaS CRDs and waiting until Established (controller creates Config after CRD is ready)... ⏳ Applying MaaS Controller CRDs from /workspace/source/test/maas-e2e/scripts/../deployment/base/maas-controller/crd... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io created customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io created ⏳ Waiting for CRD externalmodels.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io condition met ⏳ Waiting for CRD externalproviders.inference.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalproviders.inference.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io condition met ⏳ Waiting for CRD aitenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD aitenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io condition met ⏳ Waiting for CRD configs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD configs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io condition met ⏳ Waiting for CRD externalmodels.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD externalmodels.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io condition met ⏳ Waiting for CRD maasauthpolicies.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasauthpolicies.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io condition met ⏳ Waiting for CRD maasmodelrefs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maasmodelrefs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io condition met ⏳ Waiting for CRD maassubscriptions.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maassubscriptions.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io condition met ⏳ Waiting for CRD maastenantconfigs.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD maastenantconfigs.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io condition met ⏳ Waiting for CRD tenants.maas.opendatahub.io to appear (timeout: 180s)… ✅ CRD tenants.maas.opendatahub.io detected, waiting for it to become Established... customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io condition met ✅ MaaS Controller CRDs are Established [INFO] Phase 2: Applying full controller kustomize (same as operator: deployment/base/maas-controller/default)... customresourcedefinition.apiextensions.k8s.io/aitenants.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/configs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/externalproviders.inference.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/maastenantconfigs.maas.opendatahub.io configured customresourcedefinition.apiextensions.k8s.io/tenants.maas.opendatahub.io configured serviceaccount/maas-controller created role.rbac.authorization.k8s.io/maas-controller-leader-election-role created clusterrole.rbac.authorization.k8s.io/maas-controller-cluster-config-role created clusterrole.rbac.authorization.k8s.io/maas-controller-ocp-role created clusterrole.rbac.authorization.k8s.io/maas-controller-role created clusterrole.rbac.authorization.k8s.io/maas-owner-role created clusterrole.rbac.authorization.k8s.io/maas-viewer-role created rolebinding.rbac.authorization.k8s.io/maas-controller-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-cluster-config-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-ocp-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/maas-controller-rolebinding created configmap/maas-parameters created service/maas-controller-metrics created service/maas-controller-webhook-service created deployment.apps/maas-controller created servicemonitor.monitoring.coreos.com/maas-controller-metrics created networkpolicy.networking.k8s.io/maas-controller-allow-monitoring created validatingwebhookconfiguration.admissionregistration.k8s.io/maas-validating-webhook-configuration created [INFO] Restarting maas-controller to pick up manifest and ConfigMap changes deployment.apps/maas-controller restarted [INFO] Waiting for maas-controller to be ready... Waiting for deployment "maas-controller" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "maas-controller" rollout to finish: 1 old replicas are pending termination... deployment "maas-controller" successfully rolled out [INFO] Controller ready. [INFO] [INFO] Waiting for Tenant reconciler to deploy maas-api... [INFO] Applying secret migration RBAC to namespace odh-ai-gateway-infra... role.rbac.authorization.k8s.io/maas-controller-secret-migrate created rolebinding.rbac.authorization.k8s.io/maas-controller-secret-migrate created [INFO] maas-api deployment found in odh-ai-gateway-infra, waiting for rollout... Waiting for deployment "maas-api" rollout to finish: 0 of 1 updated replicas are available... deployment "maas-api" successfully rolled out [INFO] maas-api is ready [INFO] [INFO] MaaS API and MaaS Controller deployment completed successfully! [INFO] maas-api image: quay.io/opendatahub/maas-api:latest (namespace: odh-ai-gateway-infra) [INFO] maas-controller image: quay.io/opendatahub/maas-controller:latest (namespace: opendatahub) [INFO] =================================================== [INFO] Models-as-a-Service Deployment completed successfully! [INFO] =================================================== * Waiting for DataScienceCluster 'default-dsc' KServe component to be ready... * KServe (and AIGateway, if applicable) are ready in DataScienceCluster 'default-dsc' ⚠️ WARNING: Skipping Authorino readiness check (SKIP_AUTH_CHECK=true) ✅ MaaS platform deployment completed 2026-09-18T20:23:27Z deploy_platform end ---------------------------------------- Deploying Models ---------------------------------------- 2026-09-18T20:23:27Z deploy_models start Deploying MaaS system (free + premium: LLMIS + MaaSModelRef + MaaSAuthPolicy + MaaSSubscription) Waiting for Gateway openshift-ingress/maas-default-gateway to be Programmed=True (timeout: 600s)... gateway.gateway.networking.k8s.io/maas-default-gateway condition met ✅ Gateway openshift-ingress/maas-default-gateway is Programmed Creating 'llm' namespace... namespace/llm created 'models-as-a-service' namespace already exists maasauthpolicy.maas.opendatahub.io/premium-simulator-access created maasauthpolicy.maas.opendatahub.io/simulator-access created maasmodelref.maas.opendatahub.io/e2e-distinct-2-simulated created maasmodelref.maas.opendatahub.io/e2e-distinct-simulated created maasmodelref.maas.opendatahub.io/e2e-embedding-simulated created maasmodelref.maas.opendatahub.io/e2e-trlp-test-simulated created maasmodelref.maas.opendatahub.io/e2e-unconfigured-facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/facebook-opt-125m-simulated created maasmodelref.maas.opendatahub.io/premium-simulated-simulated-premium created maassubscription.maas.opendatahub.io/premium-simulator-subscription created maassubscription.maas.opendatahub.io/simulator-subscription created llminferenceservice.serving.kserve.io/e2e-distinct-2-simulated created llminferenceservice.serving.kserve.io/e2e-distinct-simulated created llminferenceservice.serving.kserve.io/e2e-embedding-simulated created llminferenceservice.serving.kserve.io/e2e-trlp-test-simulated created llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated created llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium created ✅ MaaS system deployed (free + premium + e2e test fixtures) Waiting for models to be ready (timeout: 300s)... llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated condition met llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium condition met llminferenceservice.serving.kserve.io/e2e-unconfigured-facebook-opt-125m-simulated condition met ✅ Simulator models ready Waiting for governed MaaSModelRefs to be Ready (timeout: 300s)... ✅ Governed MaaSModelRefs ready Waiting for MaaS Kuadrant AuthPolicies to be enforced (selector: app.kubernetes.io/managed-by=maas-controller, timeout: 180s)... ✅ All MaaS AuthPolicies enforced (1 policies) 2026-09-18T20:25:50Z deploy_models end Patching Authorino to log_level DEBUG... authorino.operator.authorino.kuadrant.io/authorino patched ✅ Authorino patched to log_level DEBUG Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination... deployment "authorino" successfully rolled out ---------------------------------------- Deploying ai-gateway-controller ---------------------------------------- 2026-09-18T20:25:53Z deploy_ai_gateway_controller start Deploying ai-gateway-controller image: quay.io/opendatahub/odh-ai-gateway-controller@sha256:759d42a8e41c93de070f7adc81bbc3ffeac823c7b48fa6227e005c8848a64ffc praxis-extproc image: quay.io/opendatahub/odh-praxis-extproc:odh-stable namespace: opendatahub gateway: openshift-ingress/maas-default-gateway remove maas IPP: true Opting default AITenant into praxis dataplane (maas.opendatahub.io/payload-processing-type=praxis) ... aitenant.maas.opendatahub.io/models-as-a-service annotated Pausing maas-controller (scale 1 -> 0) to avoid IPP reconcile during handoff ... deployment.apps/maas-controller scaled deployment "maas-controller" successfully rolled out Removing maas-controller legacy IPP in openshift-ingress ... deployment.apps "payload-processing" deleted destinationrule.networking.istio.io "payload-processing" deleted deployment.apps "payload-pre-processing" deleted destinationrule.networking.istio.io "payload-pre-processing" deleted envoyfilter.networking.istio.io "payload-processing" deleted networkpolicy.networking.k8s.io "payload-processing" deleted No resources found Legacy IPP Deployments removed from openshift-ingress Resuming maas-controller so ai-gateway-controller can reconcile AITenant (webhook) ... Resuming maas-controller (scale -> 1) ... deployment.apps/maas-controller scaled Waiting for deployment "maas-controller" rollout to finish: 0 of 1 updated replicas are available... deployment "maas-controller" successfully rolled out serviceaccount/ai-gateway-controller created clusterrole.rbac.authorization.k8s.io/ai-gateway-controller-role created clusterrolebinding.rbac.authorization.k8s.io/ai-gateway-controller-rolebinding created configmap/ai-gateway-controller-parameters created deployment.apps/ai-gateway-controller created Waiting for deployment "ai-gateway-controller" rollout to finish: 0 of 1 updated replicas are available... deployment "ai-gateway-controller" successfully rolled out Triggering immediate praxis-extproc install ... deployment.apps/ai-gateway-controller restarted Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "ai-gateway-controller" rollout to finish: 1 old replicas are pending termination... deployment "ai-gateway-controller" successfully rolled out Waiting for ai-gateway-controller to attach praxis finalizer on ai-tenants/models-as-a-service ... ai-gateway-controller praxis reconcile started (finalizer present) Waiting for praxis-extproc (quay.io/opendatahub/odh-praxis-extproc:odh-stable) in openshift-ingress (timeout: 300s) ... ✅ praxis-extproc ready: openshift-ingress/payload-processing image=quay.io/opendatahub/odh-praxis-extproc:odh-stable Annotating praxis IPP resources opendatahub.io/managed=false so maas-controller skips them ... deployment.apps/payload-processing annotated service/payload-processing annotated destinationrule.networking.istio.io/payload-processing annotated deployment.apps/payload-pre-processing annotated service/payload-pre-processing annotated destinationrule.networking.istio.io/payload-pre-processing annotated envoyfilter.networking.istio.io/payload-processing annotated networkpolicy.networking.k8s.io/payload-processing annotated Verified ext_proc dataplane image: quay.io/opendatahub/odh-praxis-extproc:odh-stable ai-gateway-controller rollout complete (praxis-extproc handoff done) 2026-09-18T20:26:36Z deploy_ai_gateway_controller end ---------------------------------------- Enabling tenant namespace discovery ---------------------------------------- 2026-09-18T20:26:36Z tenant_namespace_discovery start Enabling --enable-tenant-namespace-discovery on maas-controller... maas-controller already has tenant namespace discovery enabled 2026-09-18T20:26:37Z tenant_namespace_discovery end ---------------------------------------- Setting up variables for tests ---------------------------------------- -- Setting up variables for tests -- HOST: maas.apps.fff2ee93-189b-4388-ad84-9da3d9b5d7a1.prod.konfluxeaas.com MAAS_API_BASE_URL: https://maas.apps.fff2ee93-189b-4388-ad84-9da3d9b5d7a1.prod.konfluxeaas.com/maas-api ---------------------------------------- Setting up test tokens ---------------------------------------- Setting up premium test token (SA-based, works when oc whoami -t is unavailable)... Creating namespace: premium-users-namespace namespace/premium-users-namespace created Creating service account: premium-service-account serviceaccount/premium-service-account created Patching MaaSAuthPolicy premium-simulator-access to include system:serviceaccount:premium-users-namespace:premium-service-account... maasauthpolicy.maas.opendatahub.io/premium-simulator-access patched Patching MaaSSubscription premium-simulator-subscription to include system:serviceaccount:premium-users-namespace:premium-service-account... maassubscription.maas.opendatahub.io/premium-simulator-subscription patched Waiting for MaaSSubscriptions to reconcile after patch (timeout: 60s)... ✅ Both subscriptions ready: simulator-subscription=Active, premium-simulator-subscription=Active ✅ Premium test token setup complete (E2E_TEST_TOKEN_SA_* exported) Setting up test tokens (admin + regular user)... Current admin session: system:admin (will be preserved) ⚠️ No htpasswd token available - using SA token (admin tests may fail) Creating namespace: maas-admin namespace/maas-admin created Creating service account: tester-admin-user in maas-admin serviceaccount/tester-admin-user created Creating cluster role binding for tester-admin-user clusterrolebinding.rbac.authorization.k8s.io/tester-admin-user-binding created ✅ User setup completed: tester-admin-user (namespace: maas-admin) role.rbac.authorization.k8s.io/maas-admin-e2e created rolebinding.rbac.authorization.k8s.io/maas-admin-e2e-system-serviceaccount-maas-admin-tester-admin-user created auth.services.platform.opendatahub.io/auth patched ✅ Added system:serviceaccounts:maas-admin to Auth CR adminGroups (SA admin fallback) clusterrole.rbac.authorization.k8s.io/maas-admin created rolebinding.rbac.authorization.k8s.io/odh-admins-maas-admin created Creating separate SA token for regular user (required for IDOR tests)... Creating service account: tester-regular-user in default serviceaccount/tester-regular-user created Creating cluster role binding for tester-regular-user clusterrolebinding.rbac.authorization.k8s.io/tester-regular-user-binding created ✅ User setup completed: tester-regular-user (namespace: default) ✅ Regular user token for tester-regular-user (SA-based, namespace: default) Token setup complete (main session unchanged: system:admin) ---------------------------------------- Ensuring gateway allows model HTTPRoutes ---------------------------------------- /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 112: CUSTOM_RESOURCE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 113: NAMESPACE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 114: RESOURCE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 115: CRD_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 116: CSV_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 117: SUBSCRIPTION_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 118: POD_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 119: WEBHOOK_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 120: CUSTOM_CHECK_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 121: AUTHORINO_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 122: ROLLOUT_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 123: KUBECONFIG_WAIT_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 124: CATALOGSOURCE_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 125: LLMIS_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 126: MAASMODELREF_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 127: AUTHPOLICY_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 130: _MAX_TIMEOUT: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 165: LOG_LEVEL_DEBUG: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 166: LOG_LEVEL_INFO: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 167: LOG_LEVEL_WARN: readonly variable /workspace/source/test/maas-e2e/scripts/deployment-helpers.sh: line 168: LOG_LEVEL_ERROR: readonly variable ./test/e2e/scripts/prow_run_ai_gateway_controller_test.sh: line 133: derive_infra_namespace: command not found Gateway openshift-ingress/maas-default-gateway already allows HTTPRoutes from llm ---------------------------------------- Validating Deployment ---------------------------------------- 2026-09-18T20:26:40Z validate start Deployment Validation ========================================= 🚀 MaaS Platform Deployment Validation ========================================= ========================================= 1️⃣ Component Status Checks ========================================= 🔍 Checking: MaaS API pods ✅ PASS: MaaS API has 1 running pod(s) 🔍 Checking: Policy engine pods (RHCL/Kuadrant) ✅ PASS: Policy engine has 7 running pod(s) in kuadrant-system 🔍 Checking: OpenDataHub/KServe pods ℹ️ opendatahub namespace: 9 running pod(s) ✅ PASS: OpenDataHub/RHOAI has 9 total running pod(s) 🔍 Checking: LLM namespace and models ✅ PASS: Found 7 LLMInferenceService(s) with 7 running pod(s) ========================================= 2️⃣ Gateway Status ========================================= 🔍 Checking: Gateway resource ✅ PASS: Gateway is Accepted and Programmed 🔍 Checking: HTTPRoute for maas-api ✅ PASS: HTTPRoute maas-api-route is configured and accepted 🔍 Checking: Gateway hostname ✅ PASS: Gateway hostname (from MAAS_GATEWAY_HOST): https://maas.apps.fff2ee93-189b-4388-ad84-9da3d9b5d7a1.prod.konfluxeaas.com ========================================= 3️⃣ Policy Status ========================================= 🔍 Checking: AuthPolicy ✅ PASS: AuthPolicy is configured and accepted 🔍 Checking: TokenRateLimitPolicy ✅ PASS: TokenRateLimitPolicy is configured and accepted ========================================= 4️⃣ API Endpoint Tests ========================================= ℹ️ Using gateway endpoint: https://maas.apps.fff2ee93-189b-4388-ad84-9da3d9b5d7a1.prod.konfluxeaas.com 🔍 Checking: Authentication token ✅ PASS: OpenShift identity token available 🔍 Checking: MaaS API key creation ✅ PASS: MaaS API key created (name: validate-test-1789763202) 🔍 Checking: Models endpoint ℹ️ Testing: curl -sSk https://maas.apps.fff2ee93-189b-4388-ad84-9da3d9b5d7a1.prod.konfluxeaas.com/maas-api/v1/models -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" ✅ PASS: Models endpoint accessible, found 1 model(s) ℹ️ Available models: • publishers/llm/models/facebook/opt-125m - https://maas-default-gateway-openshift-default.openshift-ingress.svc.cluster.local/ ℹ️ Using first available model: publishers/llm/models/facebook/opt-125m for validation ℹ️ Rewrote internal model URL to external gateway: https://maas.apps.fff2ee93-189b-4388-ad84-9da3d9b5d7a1.prod.konfluxeaas.com/ 🔍 Checking: Model inference endpoint ℹ️ Testing: curl -sSk -X POST https://maas.apps.fff2ee93-189b-4388-ad84-9da3d9b5d7a1.prod.konfluxeaas.com/v1/chat/completions -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"model": "publishers/llm/models/facebook/opt-125m", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50}' ✅ PASS: Model inference endpoint working ℹ️ Response: {"id":"chatcmpl-dbef2490-3434-504e-9867-f28e9aaa5e4d","created":1789763203,"model":"facebook/opt-125m","usage":{"prompt_tokens":6,"completion_tokens":7,"total_tokens":13},"object":"chat.completion","k 🔍 Checking: Rate limiting ℹ️ User tier: unknown (could not extract from token) ℹ️ Sending 10 rapid requests to test rate limiting... ✅ PASS: Rate limiting is working (5 successful, 5 rate limited) 🔍 Checking: Authorization enforcement (401 without token) ✅ PASS: Authorization is enforced (got 401 without token) ========================================= 📊 Validation Summary ========================================= Results: ✅ Passed: 15 ❌ Failed: 0 ⚠️ Warnings: 0 ✅ PASS: All critical checks passed! 🎉 Next steps: 1. Deploy a model: kustomize build docs/samples/models/simulator | kubectl apply -f - 2. Access the API at: https://maas.apps.fff2ee93-189b-4388-ad84-9da3d9b5d7a1.prod.konfluxeaas.com} 3. Check documentation: docs/README.md 4. Re-run validation with specific model: ./scripts/validate-deployment.sh MODEL_NAME Deployment validation completed 2026-09-18T20:26:43Z validate end ---------------------------------------- Running E2E Tests ---------------------------------------- -- E2E Tests (ai-gateway-controller, no external-model tests) -- Waiting for gateway: https://maas.apps.fff2ee93-189b-4388-ad84-9da3d9b5d7a1.prod.konfluxeaas.com/maas-api/health ... Gateway reachable (HTTP 200) Waiting for authenticated gateway access ... Authenticated gateway access working (HTTP 200) Creating Python venv for e2e tests... WARNING: The directory '/opt/app-root/src/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. Requirement already satisfied: pip in ./test/e2e/.venv/lib/python3.9/site-packages (21.3.1) Collecting pip Downloading pip-26.0.1-py3-none-any.whl (1.8 MB) Requirement already satisfied: setuptools in ./test/e2e/.venv/lib/python3.9/site-packages (53.0.0) Collecting setuptools Downloading setuptools-82.0.1-py3-none-any.whl (1.0 MB) Installing collected packages: setuptools, pip Attempting uninstall: setuptools Found existing installation: setuptools 53.0.0 Uninstalling setuptools-53.0.0: Successfully uninstalled setuptools-53.0.0 Attempting uninstall: pip Found existing installation: pip 21.3.1 Uninstalling pip-21.3.1: Successfully uninstalled pip-21.3.1 Successfully installed pip-26.0.1 setuptools-82.0.1 WARNING: The directory '/opt/app-root/src/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. WARNING: The directory '/opt/app-root/src/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. Running E2E pass 1/2: parallel (E2E_PARALLEL_WORKERS=7, --dist=loadgroup, -m 'not serial') ============================= test session starts ============================== platform linux -- Python 3.9.25, pytest-8.4.2, pluggy-1.6.0 -- /workspace/source/test/e2e/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.9.25', 'Platform': 'Linux-5.14.0-570.134.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '8.4.2', 'pluggy': '1.6.0'}, 'Plugins': {'xdist': '3.8.0', 'metadata': '3.1.1', 'html': '4.2.0'}, 'PLATFORM': 'el9'} rootdir: /workspace/source/test/maas-e2e/test/e2e configfile: pyproject.toml plugins: xdist-3.8.0, metadata-3.1.1, html-4.2.0 created: 7/7 workers 7 workers [211 items] scheduling tests via LoadGroupScheduling test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_default_aitenant_bootstraps_maas_tenant_config_without_gateway_mutation@mt_lifecycle test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_explicit_subscription_header@models test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_model_routes_through_tenant_gateway@tenant_isolation test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_create_api_key@api_keys test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_subscription_namespace_visible_to_api@security test/maas-e2e/test/e2e/tests/test_smoke.py::test_healthz_or_404@readonly [gw4] [ 0%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_subscription_namespace_visible_to_api@security test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_another_namespace_not_visible_to_api@security [gw4] [ 0%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSAPIWatchNamespace::test_subscription_in_another_namespace_not_visible_to_api@security test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_maas_subscription_namespace@security [gw4] [ 1%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_maas_subscription_namespace@security test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_another_namespace@security [gw4] [ 1%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestMaaSControllerWatchNamespace::test_authpolicy_and_subscription_in_another_namespace@security test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_auth_policy_model_ref@security [gw4] [ 2%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_auth_policy_model_ref@security test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_subscription_model_ref@security [gw4] [ 2%] SKIPPED test/maas-e2e/test/e2e/tests/test_namespace_scoping.py::TestModelRef::test_subscription_model_ref@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAPIKeyManagementIsolation::test_api_key_cannot_mint_another_api_key@security [gw5] [ 3%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_healthz_or_404@readonly test/maas-e2e/test/e2e/tests/test_smoke.py::test_tokens_endpoint_replaced_by_api_keys@readonly [gw5] [ 3%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_tokens_endpoint_replaced_by_api_keys@readonly test/maas-e2e/test/e2e/tests/test_smoke.py::test_models_catalog@readonly [gw5] [ 4%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_models_catalog@readonly test/maas-e2e/test/e2e/tests/test_smoke.py::test_chat_completions_gateway_alive@readonly [gw5] [ 4%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_chat_completions_gateway_alive@readonly test/maas-e2e/test/e2e/tests/test_smoke.py::test_legacy_completions_optionally@readonly [gw5] [ 5%] PASSED test/maas-e2e/test/e2e/tests/test_smoke.py::test_legacy_completions_optionally@readonly test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_tenant_ready_and_phase_healthy@readonly [gw4] [ 5%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAPIKeyManagementIsolation::test_api_key_cannot_mint_another_api_key@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[username-only]@security [gw4] [ 6%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[username-only]@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[group-only]@security [gw4] [ 6%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[group-only]@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[keyname-only]@security [gw5] [ 7%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_tenant_ready_and_phase_healthy@readonly test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_status_has_phase_and_conditions@readonly [gw4] [ 7%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_forged_identity_headers_rejected_on_key_mint[keyname-only]@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_injected_identity_headers_rejected_on_inference@security [gw5] [ 8%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_status_has_phase_and_conditions@readonly test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_spec_is_well_formed@readonly [gw4] [ 8%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_injected_identity_headers_rejected_on_inference@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_duplicate_subscription_headers_ignored@security [gw5] [ 9%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_spec_is_well_formed@readonly test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_conditions_use_kubernetes_metav1_shape@readonly [gw5] [ 9%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantContract::test_conditions_use_kubernetes_metav1_shape@readonly test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantNoFalseOwnership::test_maas_user_crs_not_owned_by_tenant@readonly [gw5] [ 9%] PASSED test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantNoFalseOwnership::test_maas_user_crs_not_owned_by_tenant@readonly test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_default_exists@readonly [gw5] [ 10%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_default_exists@readonly test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_not_terminating@readonly [gw1] [ 10%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_default_aitenant_bootstraps_maas_tenant_config_without_gateway_mutation@mt_lifecycle test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_rejected_outside_ai_tenants_namespace@mt_lifecycle [gw5] [ 11%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigAnchorPresence::test_cluster_config_not_terminating@readonly test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_default_aitenant_lists_config_owner_reference@readonly [gw5] [ 11%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_default_aitenant_lists_config_owner_reference@readonly test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_tenant_config_lists_config_owner_reference@readonly [gw5] [ 12%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_tenant_config_lists_config_owner_reference@readonly test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_maas_controller_deployment_does_not_list_config_owner_reference@readonly [gw5] [ 12%] PASSED test/maas-e2e/test/e2e/tests/test_config_tenant.py::TestConfigTenantOwnership::test_maas_controller_deployment_does_not_list_config_owner_reference@readonly test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_requires_auth@readonly [gw5] [ 13%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_requires_auth@readonly test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_with_invalid_token@readonly [gw5] [ 13%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_with_invalid_token@readonly test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_authenticated@readonly [gw5] [ 14%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_authenticated@readonly test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_gateway_matches_deployment@readonly [gw5] [ 14%] SKIPPED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_gateway_matches_deployment@readonly test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_not_exposed_through_gateway@readonly [gw5] [ 15%] PASSED test/maas-e2e/test/e2e/tests/test_tenant_discovery.py::test_tenant_discovery_not_exposed_through_gateway@readonly [gw4] [ 15%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderSpoofing::test_duplicate_subscription_headers_ignored@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestExpiredKeyRejection::test_expired_key_rejected_at_gateway@security [gw4] [ 16%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestExpiredKeyRejection::test_expired_key_rejected_at_gateway@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestCrossModelAccess::test_key_cannot_access_model_outside_subscription@security [gw4] [ 16%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestCrossModelAccess::test_key_cannot_access_model_outside_subscription@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_subscription_with_nonexistent_model_ref@security [gw4] [ 17%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_subscription_with_nonexistent_model_ref@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_authpolicy_with_nonexistent_model_ref@security [gw4] [ 17%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestMissingModelRef::test_authpolicy_with_nonexistent_model_ref@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderAbuse::test_special_characters_in_subscription_header@security [gw4] [ 18%] PASSED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestHeaderAbuse::test_special_characters_in_subscription_header@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestWebhookValidation::test_subscription_rejected_in_unlabeled_namespace@security [gw4] [ 18%] FAILED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestWebhookValidation::test_subscription_rejected_in_unlabeled_namespace@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestWebhookValidation::test_authpolicy_rejected_in_unlabeled_namespace@security [gw4] [ 18%] FAILED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestWebhookValidation::test_authpolicy_rejected_in_unlabeled_namespace@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[subscriptions-select]@security [gw1] [ 19%] PASSED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_rejected_outside_ai_tenants_namespace@mt_lifecycle test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_create_bootstrap_resources@mt_lifecycle [gw4] [ 19%] FAILED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[subscriptions-select]@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-cleanup]@security [gw1] [ 20%] FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_create_bootstrap_resources@mt_lifecycle test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_migrates_and_removes_legacy_tenant@mt_lifecycle [gw4] [ 20%] FAILED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-cleanup]@security test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-validate]@security [gw1] [ 21%] FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_migrates_and_removes_legacy_tenant@mt_lifecycle test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects@mt_lifecycle [gw0] [ 21%] ERROR test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_create_api_key@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_list_api_keys@api_keys [gw3] [ 22%] ERROR test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_explicit_subscription_header@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_empty_subscription_header_value@models [gw0] [ 22%] ERROR test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_list_api_keys@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_revoke_api_key@api_keys [gw3] [ 23%] ERROR test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_empty_subscription_header_value@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_models_filtered_by_subscription@models [gw0] [ 23%] ERROR test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_revoke_api_key@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_admin_manage_other_users_keys@api_keys [gw3] [ 24%] ERROR test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_models_filtered_by_subscription@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_deduplication_same_model_multiple_refs@models [gw0] [ 24%] ERROR test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_admin_manage_other_users_keys@api_keys test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_non_admin_cannot_access_other_users_keys@api_keys [gw3] [ 25%] ERROR test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_deduplication_same_model_multiple_refs@models test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_distinct_models_in_subscription@models [gw0] [ 25%] ERROR test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_non_admin_cannot_access_other_users_keys@api_keys [gw3] [ 26%] ERROR test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_distinct_models_in_subscription@models [gw4] [ 26%] FAILED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-validate]@security [gw1] [ 27%] FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects@mt_lifecycle test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_derives_non_default_tenant_namespace@mt_lifecycle [gw2] [ 27%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_model_routes_through_tenant_gateway@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_inference_succeeds_through_tenant_gateway@tenant_isolation [gw2] [ 27%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_inference_succeeds_through_tenant_gateway@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_tenant_isolation_cross_gateway_blocked@tenant_isolation [gw2] [ 28%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_tenant_isolation_cross_gateway_blocked@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_correct_model_in_body_succeeds@tenant_isolation [gw2] [ 28%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_correct_model_in_body_succeeds@tenant_isolation test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_wrong_model_in_body_rejected@tenant_isolation [gw2] [ 29%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_wrong_model_in_body_rejected@tenant_isolation [gw1] [ 29%] FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_derives_non_default_tenant_namespace@mt_lifecycle test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_labeled_tenant_namespace_is_discovered@mt_lifecycle [gw1] [ 30%] ERROR test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_labeled_tenant_namespace_is_discovered@mt_lifecycle ==================================== ERRORS ==================================== _____________ ERROR at setup of TestAPIKeyCRUD.test_create_api_key _____________ [gw0] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python request = > @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/maas-e2e/test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/maas-e2e/test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/maas-e2e/test/e2e/tests/worker_tenant_fixtures.py:235: in bootstrap_worker_tenant bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-worker-w0-6ddb93', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-worker-w0-6ddb93 -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError ---------------------------- Captured stdout setup ----------------------------- [token] using env TOKEN (masked): 1016 [cleanup] failed to delete aitenant/e2e-worker-w0-6ddb93: `oc delete aitenant e2e-worker-w0-6ddb93 --ignore-not-found --timeout=180s -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete namespace/ai-tenant-e2e-worker-w0-6ddb93: `oc delete namespace ai-tenant-e2e-worker-w0-6ddb93 --ignore-not-found --timeout=90s` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete route/e2e-worker-w0-6ddb93-route: `oc delete route e2e-worker-w0-6ddb93-route --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete gateway/e2e-worker-w0-6ddb93: `oc delete gateway e2e-worker-w0-6ddb93 --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete configmap/e2e-worker-w0-6ddb93-gw-options: `oc delete configmap e2e-worker-w0-6ddb93-gw-options --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to remove gateway access label for e2e-worker-w0-6ddb93: failed to remove gateway access label from odh-ai-gateway-infra: Unable to connect to the server: net/http: TLS handshake timeout ------------------------------ Captured log setup ------------------------------ WARNING test_helper:test_helper.py:630 Failed to delete MaaSAuthPolicy/simulator-access in ai-tenant-e2e-worker-w0-6ddb93: Unable to connect to the server: net/http: TLS handshake timeout WARNING test_helper:test_helper.py:630 Failed to delete MaaSSubscription/simulator-subscription in ai-tenant-e2e-worker-w0-6ddb93: Unable to connect to the server: net/http: TLS handshake timeout ____ ERROR at setup of TestModelsEndpoint.test_explicit_subscription_header ____ [gw3] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python request = > @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/maas-e2e/test/e2e/tests/test_models_endpoint.py:92: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/maas-e2e/test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/maas-e2e/test/e2e/tests/worker_tenant_fixtures.py:235: in bootstrap_worker_tenant bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-worker-w3-74bb52', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-worker-w3-74bb52 -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError ---------------------------- Captured stdout setup ----------------------------- [cleanup] failed to delete aitenant/e2e-worker-w3-74bb52: `oc delete aitenant e2e-worker-w3-74bb52 --ignore-not-found --timeout=180s -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete namespace/ai-tenant-e2e-worker-w3-74bb52: `oc delete namespace ai-tenant-e2e-worker-w3-74bb52 --ignore-not-found --timeout=90s` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete route/e2e-worker-w3-74bb52-route: `oc delete route e2e-worker-w3-74bb52-route --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete gateway/e2e-worker-w3-74bb52: `oc delete gateway e2e-worker-w3-74bb52 --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete configmap/e2e-worker-w3-74bb52-gw-options: `oc delete configmap e2e-worker-w3-74bb52-gw-options --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to remove gateway access label for e2e-worker-w3-74bb52: failed to remove gateway access label from odh-ai-gateway-infra: Unable to connect to the server: net/http: TLS handshake timeout ------------------------------ Captured log setup ------------------------------ WARNING test_helper:test_helper.py:630 Failed to delete MaaSAuthPolicy/simulator-access in ai-tenant-e2e-worker-w3-74bb52: Unable to connect to the server: net/http: TLS handshake timeout WARNING test_helper:test_helper.py:630 Failed to delete MaaSSubscription/simulator-subscription in ai-tenant-e2e-worker-w3-74bb52: Unable to connect to the server: net/http: TLS handshake timeout _____________ ERROR at setup of TestAPIKeyCRUD.test_list_api_keys ______________ [gw0] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python request = > @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/maas-e2e/test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/maas-e2e/test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/maas-e2e/test/e2e/tests/worker_tenant_fixtures.py:235: in bootstrap_worker_tenant bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-worker-w0-6ddb93', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-worker-w0-6ddb93 -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError __ ERROR at setup of TestModelsEndpoint.test_empty_subscription_header_value ___ [gw3] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python request = > @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/maas-e2e/test/e2e/tests/test_models_endpoint.py:92: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/maas-e2e/test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/maas-e2e/test/e2e/tests/worker_tenant_fixtures.py:235: in bootstrap_worker_tenant bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-worker-w3-74bb52', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-worker-w3-74bb52 -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError _____________ ERROR at setup of TestAPIKeyCRUD.test_revoke_api_key _____________ [gw0] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python request = > @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/maas-e2e/test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/maas-e2e/test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/maas-e2e/test/e2e/tests/worker_tenant_fixtures.py:235: in bootstrap_worker_tenant bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-worker-w0-6ddb93', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-worker-w0-6ddb93 -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError __ ERROR at setup of TestModelsEndpoint.test_models_filtered_by_subscription ___ [gw3] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python request = > @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/maas-e2e/test/e2e/tests/test_models_endpoint.py:92: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/maas-e2e/test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/maas-e2e/test/e2e/tests/worker_tenant_fixtures.py:235: in bootstrap_worker_tenant bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-worker-w3-74bb52', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-worker-w3-74bb52 -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError _ ERROR at setup of TestAPIKeyAuthorization.test_admin_manage_other_users_keys _ [gw0] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python request = > @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/maas-e2e/test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/maas-e2e/test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/maas-e2e/test/e2e/tests/worker_tenant_fixtures.py:235: in bootstrap_worker_tenant bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-worker-w0-6ddb93', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-worker-w0-6ddb93 -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError _ ERROR at setup of TestModelsEndpoint.test_deduplication_same_model_multiple_refs _ [gw3] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python request = > @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/maas-e2e/test/e2e/tests/test_models_endpoint.py:92: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/maas-e2e/test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/maas-e2e/test/e2e/tests/worker_tenant_fixtures.py:235: in bootstrap_worker_tenant bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-worker-w3-74bb52', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-worker-w3-74bb52 -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError _ ERROR at setup of TestAPIKeyAuthorization.test_non_admin_cannot_access_other_users_keys _ [gw0] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python request = > @pytest.fixture(scope="module", autouse=True) def _worker_api_keys_context(request): """Bind parallel API-key tests to worker state; leave the serial pass alone.""" from worker_tenant_fixtures import activate_worker_tenant, serial_only_selection if serial_only_selection(request): yield None return > context = request.getfixturevalue("worker_tenant_context") test/maas-e2e/test/e2e/tests/test_api_keys.py:84: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/maas-e2e/test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/maas-e2e/test/e2e/tests/worker_tenant_fixtures.py:235: in bootstrap_worker_tenant bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-worker-w0-6ddb93', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-worker-w0-6ddb93 -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError _ ERROR at setup of TestModelsEndpoint.test_multiple_distinct_models_in_subscription _ [gw3] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python request = > @pytest.fixture(scope="module", autouse=True) def _worker_models_context(request): """Route parallel model tests through the explicit worker context. The serial pass intentionally retains the default deployment. """ from worker_tenant_fixtures import ( activate_worker_tenant, ensure_worker_models, serial_only_selection, ) if serial_only_selection(request): yield return > context = request.getfixturevalue("worker_tenant_context") test/maas-e2e/test/e2e/tests/test_models_endpoint.py:92: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:549: in getfixturevalue fixturedef = self._get_active_fixturedef(argname) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:640: in _get_active_fixturedef fixturedef.execute(request=subrequest) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1128: in execute result = ihook.pytest_fixture_setup(fixturedef=self, request=request) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_hooks.py:512: in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/pluggy/_manager.py:120: in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/setuponly.py:36: in pytest_fixture_setup return (yield) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:1196: in pytest_fixture_setup result = call_fixture_func(fixturefunc, request, kwargs) test/e2e/.venv/lib64/python3.9/site-packages/_pytest/fixtures.py:923: in call_fixture_func fixture_result = next(generator) test/maas-e2e/test/e2e/tests/conftest.py:37: in worker_tenant_context case = bootstrap_worker_tenant(case) test/maas-e2e/test/e2e/tests/worker_tenant_fixtures.py:235: in bootstrap_worker_tenant bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-worker-w3-74bb52', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-worker-w3-74bb52 -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError _ ERROR at setup of TestTenantModelInference.test_model_routes_through_tenant_gateway _ [gw2] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:85: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-26ffff', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-inf-a-26ffff -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError ---------------------------- Captured stdout setup ----------------------------- [cleanup] failed to delete aitenant/e2e-inf-a-26ffff: `oc delete aitenant e2e-inf-a-26ffff --ignore-not-found --timeout=180s -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete namespace/ai-tenant-e2e-inf-a-26ffff: `oc delete namespace ai-tenant-e2e-inf-a-26ffff --ignore-not-found --timeout=90s` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete route/e2e-inf-a-26ffff-route: `oc delete route e2e-inf-a-26ffff-route --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete gateway/e2e-inf-a-26ffff: `oc delete gateway e2e-inf-a-26ffff --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete configmap/e2e-inf-a-26ffff-gw-options: `oc delete configmap e2e-inf-a-26ffff-gw-options --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to remove gateway access label for e2e-inf-a-26ffff: failed to remove gateway access label from odh-ai-gateway-infra: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete aitenant/e2e-inf-b-2ceaf7: `oc delete aitenant e2e-inf-b-2ceaf7 --ignore-not-found --timeout=180s -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete namespace/ai-tenant-e2e-inf-b-2ceaf7: `oc delete namespace ai-tenant-e2e-inf-b-2ceaf7 --ignore-not-found --timeout=90s` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete gateway/e2e-inf-b-2ceaf7: `oc delete gateway e2e-inf-b-2ceaf7 --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: http2: client connection lost [cleanup] failed to delete configmap/e2e-inf-b-2ceaf7-gw-options: `oc delete configmap e2e-inf-b-2ceaf7-gw-options --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to remove gateway access label for e2e-inf-b-2ceaf7: failed to remove gateway access label from odh-ai-gateway-infra: Unable to connect to the server: net/http: TLS handshake timeout ------------------------------ Captured log setup ------------------------------ WARNING test_helper:test_helper.py:630 Failed to delete MaaSAuthPolicy/e2e-inf-a-26ffff-policy in ai-tenant-e2e-inf-a-26ffff: Unable to connect to the server: net/http: TLS handshake timeout WARNING test_helper:test_helper.py:630 Failed to delete MaaSSubscription/e2e-inf-a-26ffff-sub in ai-tenant-e2e-inf-a-26ffff: Unable to connect to the server: net/http: TLS handshake timeout WARNING test_helper:test_helper.py:630 Failed to delete MaaSAuthPolicy/e2e-inf-b-2ceaf7-policy in ai-tenant-e2e-inf-b-2ceaf7: Unable to connect to the server: net/http: TLS handshake timeout WARNING test_helper:test_helper.py:630 Failed to delete MaaSSubscription/e2e-inf-b-2ceaf7-sub in ai-tenant-e2e-inf-b-2ceaf7: Unable to connect to the server: net/http: TLS handshake timeout _ ERROR at setup of TestTenantModelInference.test_inference_succeeds_through_tenant_gateway _ [gw2] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:85: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-26ffff', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-inf-a-26ffff -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError _ ERROR at setup of TestTenantModelInference.test_tenant_isolation_cross_gateway_blocked _ [gw2] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:85: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-26ffff', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-inf-a-26ffff -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError _ ERROR at setup of TestTenantBodyRouting.test_correct_model_in_body_succeeds __ [gw2] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:85: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-26ffff', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-inf-a-26ffff -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError __ ERROR at setup of TestTenantBodyRouting.test_wrong_model_in_body_rejected ___ [gw2] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module") def tenant_inference_cases(): """Set up two tenants with models for inference testing.""" require_aitenant_crd() case_a = new_named_tenant_case("e2e-inf-a") case_b = new_named_tenant_case("e2e-inf-b") try: # Bootstrap tenants for case in (case_a, case_b): > bootstrap_aitenant_tenant(case) test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py:85: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:838: in bootstrap_aitenant_tenant wait_for_json(AITENANT_KIND, case["tenant_label_name"], AITENANT_NAMESPACE, predicate=aitenant_ready) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:223: in wait_for_json obj = get_json_or_none(kind, name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'aitenant', name = 'e2e-inf-a-26ffff', namespace = 'ai-tenants' def get_json_or_none(kind: str, name: str, namespace: Optional[str] = None) -> Optional[dict]: args = ["get", kind, name, "-o", "json"] if namespace: args.extend(["-n", namespace]) result = _oc_run(args) if result.returncode == 0: return json.loads(result.stdout) if _oc_output_not_found(result): return None > raise RuntimeError(f"`oc {' '.join(args)}` failed: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: `oc get aitenant e2e-inf-a-26ffff -o json -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:194: RuntimeError _ ERROR at setup of TestTenantNamespaceDiscovery.test_labeled_tenant_namespace_is_discovered _ [gw1] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python @pytest.fixture(scope="module", autouse=True) def _require_multitenancy_prerequisites(): > require_tenant_namespace_discovery() test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py:73: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:407: in require_tenant_namespace_discovery if not controller_has_tenant_namespace_discovery(): test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:365: in controller_has_tenant_namespace_discovery result = _oc_run(["get", "deployment", "maas-controller", "-n", DEPLOYMENT_NAMESPACE, "-o", "json"]) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:86: in _oc_run return subprocess.run( /usr/lib64/python3.9/subprocess.py:507: in run stdout, stderr = process.communicate(input, timeout=timeout) /usr/lib64/python3.9/subprocess.py:1134: in communicate stdout, stderr = self._communicate(input, endtime, timeout) /usr/lib64/python3.9/subprocess.py:1996: in _communicate self._check_timeout(endtime, orig_timeout, stdout, stderr) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = endtime = 114956.716666878, orig_timeout = 60, stdout_seq = [], stderr_seq = [] skip_check_and_raise = False def _check_timeout(self, endtime, orig_timeout, stdout_seq, stderr_seq, skip_check_and_raise=False): """Convenience for checking if a timeout has expired.""" if endtime is None: return if skip_check_and_raise or _time() > endtime: > raise TimeoutExpired( self.args, orig_timeout, output=b''.join(stdout_seq) if stdout_seq else None, stderr=b''.join(stderr_seq) if stderr_seq else None) E subprocess.TimeoutExpired: Command '['/usr/local/bin/oc', 'get', 'deployment', 'maas-controller', '-n', 'opendatahub', '-o', 'json']' timed out after 60 seconds /usr/lib64/python3.9/subprocess.py:1178: TimeoutExpired =================================== FAILURES =================================== ___ TestWebhookValidation.test_subscription_rejected_in_unlabeled_namespace ____ [gw4] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python self = def test_subscription_rejected_in_unlabeled_namespace(self): """MaaSSubscription create is rejected in namespace without MaasTenantConfig CR. Webhooks require namespaces to have a MaasTenantConfig CR to contain tenant resources. """ test_ns = f"e2e-webhook-test-{uuid.uuid4().hex[:6]}" try: # Create namespace without MaasTenantConfig CR result = subprocess.run( ["oc", "create", "namespace", test_ns], capture_output=True, text=True, timeout=30 ) assert result.returncode == 0, f"Failed to create namespace: {result.stderr}" # Try to create MaaSSubscription (should be rejected by webhook) result = subprocess.run( ["oc", "apply", "-f", "-"], input=json.dumps({ "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSSubscription", "metadata": {"name": "test-sub", "namespace": test_ns}, "spec": { "owner": {"groups": [{"name": "system:authenticated"}]}, "modelRefs": [{ "name": MODEL_REF, "namespace": MODEL_NAMESPACE, "tokenRateLimits": [{"limit": 100, "window": "1m"}] }], }, }), capture_output=True, text=True, timeout=30 ) # Verify webhook rejection assert result.returncode != 0, "Expected webhook to reject subscription in namespace without MaasTenantConfig CR" assert "admission webhook" in result.stderr.lower(), \ f"Expected webhook rejection, got: {result.stderr}" assert "not enabled for MaaS tenant resources" in result.stderr, \ f"Expected helpful error message, got: {result.stderr}" assert "Create a MaasTenantConfig CR" in result.stderr, \ f"Expected error to mention creating MaasTenantConfig CR, got: {result.stderr}" log.info("✅ Webhook correctly rejected MaaSSubscription in namespace without MaasTenantConfig CR") log.info(f"Error message: {result.stderr}") finally: # Clean up namespace > subprocess.run( ["oc", "delete", "namespace", test_ns, "--ignore-not-found"], capture_output=True, text=True, timeout=30 ) test/maas-e2e/test/e2e/tests/test_negative_security.py:672: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python3.9/subprocess.py:507: in run stdout, stderr = process.communicate(input, timeout=timeout) /usr/lib64/python3.9/subprocess.py:1134: in communicate stdout, stderr = self._communicate(input, endtime, timeout) /usr/lib64/python3.9/subprocess.py:1996: in _communicate self._check_timeout(endtime, orig_timeout, stdout, stderr) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = endtime = 114575.603695584, orig_timeout = 30 stdout_seq = [b'namespace "e2e-webhook-test-2dc7de" deleted\n'], stderr_seq = [] skip_check_and_raise = False def _check_timeout(self, endtime, orig_timeout, stdout_seq, stderr_seq, skip_check_and_raise=False): """Convenience for checking if a timeout has expired.""" if endtime is None: return if skip_check_and_raise or _time() > endtime: > raise TimeoutExpired( self.args, orig_timeout, output=b''.join(stdout_seq) if stdout_seq else None, stderr=b''.join(stderr_seq) if stderr_seq else None) E subprocess.TimeoutExpired: Command '['oc', 'delete', 'namespace', 'e2e-webhook-test-2dc7de', '--ignore-not-found']' timed out after 30 seconds /usr/lib64/python3.9/subprocess.py:1178: TimeoutExpired ------------------------------ Captured log call ------------------------------- INFO test_negative_security:test_negative_security.py:667 ✅ Webhook correctly rejected MaaSSubscription in namespace without MaasTenantConfig CR INFO test_negative_security:test_negative_security.py:668 Error message: Error from server (Forbidden): error when creating "STDIN": admission webhook "vmaassubscription.kb.io" denied the request: namespace "e2e-webhook-test-2dc7de" is not enabled for MaaS tenant resources. Create a MaasTenantConfig CR in this namespace to enable it. ____ TestWebhookValidation.test_authpolicy_rejected_in_unlabeled_namespace _____ [gw4] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python self = def test_authpolicy_rejected_in_unlabeled_namespace(self): """MaaSAuthPolicy create is rejected in namespace without MaasTenantConfig CR.""" test_ns = f"e2e-webhook-test-{uuid.uuid4().hex[:6]}" try: # Create namespace without MaasTenantConfig CR result = subprocess.run( ["oc", "create", "namespace", test_ns], capture_output=True, text=True, timeout=30 ) assert result.returncode == 0, f"Failed to create namespace: {result.stderr}" # Try to create MaaSAuthPolicy (should be rejected by webhook) result = subprocess.run( ["oc", "apply", "-f", "-"], input=json.dumps({ "apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSAuthPolicy", "metadata": {"name": "test-policy", "namespace": test_ns}, "spec": { "modelRefs": [{"name": MODEL_REF, "namespace": MODEL_NAMESPACE}], "subjects": {"groups": [{"name": "system:authenticated"}]}, }, }), capture_output=True, text=True, timeout=30 ) # Verify webhook rejection assert result.returncode != 0, "Expected webhook to reject auth policy in namespace without MaasTenantConfig CR" > assert "admission webhook" in result.stderr.lower(), \ f"Expected webhook rejection, got: {result.stderr}" E AssertionError: Expected webhook rejection, got: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": EOF E E assert 'admission webhook' in 'error from server (internalerror): error when creating "stdin": internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": eof\n' E + where 'error from server (internalerror): error when creating "stdin": internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": eof\n' = () E + where = 'Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": EOF\n'.lower E + where 'Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": EOF\n' = CompletedProcess(args=['oc', 'apply', '-f', '-'], returncode=1, stdout='', stderr='Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaasauthpolicy.kb.io": failed to call webhook: Post "https://maas-controller-webhook-service.opendatahub.svc:443/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": EOF\n').stderr test/maas-e2e/test/e2e/tests/test_negative_security.py:706: AssertionError _ TestInternalEndpointIsolation.test_internal_endpoint_not_routable[subscriptions-select] _ [gw4] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python self = path = '/internal/v1/subscriptions/select' body = {'groups': ['system:authenticated'], 'username': 'test'} @pytest.mark.parametrize( "path,body", [ ( "/internal/v1/subscriptions/select", {"groups": ["system:authenticated"], "username": "test"}, ), ("/internal/v1/api-keys/cleanup", {}), ("/internal/v1/api-keys/validate", {"key": "sk-oai-test-fake-key"}), ], ids=["subscriptions-select", "api-keys-cleanup", "api-keys-validate"], ) def test_internal_endpoint_not_routable(self, path, body): """POST /maas-api/internal/* through gateway must not reach the handler. Sends an authenticated request to each internal endpoint via the gateway. With the HTTPRoute scoped to /maas-api/v1, these paths have no matching route and the gateway returns a non-success status (typically 404). The response must not contain handler output. """ > _wait_for_gateway_auth_enforced() test/maas-e2e/test/e2e/tests/test_negative_security.py:757: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_helper.py:1222: in _wait_for_gateway_auth_enforced cr = _get_cr("authpolicy", name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'authpolicy', name = 'maas-gateway-auth', namespace = 'openshift-ingress' def _get_cr(kind, name, namespace=None): """Get a CR as dict, or None if not found. Retries on transient errors. Returns None only when the resource genuinely does not exist (server NotFound). Raises RuntimeError for other failures (RBAC, missing CRD, transport errors that persist after retries) so callers can distinguish infrastructure issues from true absence. """ namespace = namespace or _ns() max_retries = 3 retry_delay = 2 for attempt in range(max_retries): result = subprocess.run(["oc", "get", kind, name, "-n", namespace, "-o", "json"], capture_output=True, text=True) if result.returncode == 0: return json.loads(result.stdout) if attempt < max_retries - 1 and _is_transient_kubectl_error(result.stderr): log.warning( f"Transient kubectl error getting {kind}/{name} (attempt {attempt + 1}/{max_retries}): {result.stderr.strip()}" ) time.sleep(retry_delay * (attempt + 1)) continue # Terminal failure — distinguish not-found from other errors if _is_not_found_error(result.stderr): return None log.error( f"Failed to get {kind}/{name} in namespace '{namespace}' after {attempt + 1} attempts. " f"Last error: {result.stderr.strip()}" ) > raise RuntimeError( f"Failed to get {kind}/{name} in namespace '{namespace}': {result.stderr.strip()}" ) E RuntimeError: Failed to get authpolicy/maas-gateway-auth in namespace 'openshift-ingress': Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/test_helper.py:688: RuntimeError ------------------------------ Captured log call ------------------------------- INFO test_helper:test_helper.py:1214 Waiting for gateway AuthPolicy openshift-ingress/maas-gateway-auth Accepted+Enforced (timeout: 240s)... WARNING test_helper:test_helper.py:674 Transient kubectl error getting authpolicy/maas-gateway-auth (attempt 1/3): Unable to connect to the server: net/http: TLS handshake timeout WARNING test_helper:test_helper.py:674 Transient kubectl error getting authpolicy/maas-gateway-auth (attempt 2/3): Unable to connect to the server: net/http: TLS handshake timeout ERROR test_helper:test_helper.py:684 Failed to get authpolicy/maas-gateway-auth in namespace 'openshift-ingress' after 3 attempts. Last error: Unable to connect to the server: net/http: TLS handshake timeout ________ TestAITenantLifecycle.test_aitenant_create_bootstrap_resources ________ [gw1] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python self = def test_aitenant_create_bootstrap_resources(self): case = _new_aitenant_case() try: > _apply_gateway_fixture(case) test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:420: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:183: in _apply_gateway_fixture apply_https_gateway_fixture( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:548: in apply_gateway_fixture apply_gateway_access_label(INFRA_NAMESPACE, gateway_name) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:468: in apply_gateway_access_label ensure_namespace(namespace, labels={gateway_access_label_key(gateway_name): "true"}) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'odh-ai-gateway-infra' def ensure_namespace(name: str, *, labels: Optional[dict[str, str]] = None) -> None: result = _oc_run(["create", "namespace", name]) if result.returncode != 0 and "AlreadyExists" not in (result.stderr or "") and "already exists" not in (result.stderr or "").lower(): > raise RuntimeError(f"failed to create namespace {name}: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: failed to create namespace odh-ai-gateway-infra: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:481: RuntimeError ----------------------------- Captured stdout call ----------------------------- [cleanup] failed to delete aitenant/e2e-ait-951d59c0: `oc delete aitenant e2e-ait-951d59c0 --ignore-not-found --timeout=180s -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete gateway/e2e-ait-951d59c0: `oc delete gateway e2e-ait-951d59c0 --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete configmap/e2e-ait-951d59c0-gw-options: `oc delete configmap e2e-ait-951d59c0-gw-options --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to remove gateway access label for e2e-ait-951d59c0: failed to remove gateway access label from odh-ai-gateway-infra: Unable to connect to the server: net/http: TLS handshake timeout _ TestInternalEndpointIsolation.test_internal_endpoint_not_routable[api-keys-cleanup] _ [gw4] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python self = path = '/internal/v1/api-keys/cleanup', body = {} @pytest.mark.parametrize( "path,body", [ ( "/internal/v1/subscriptions/select", {"groups": ["system:authenticated"], "username": "test"}, ), ("/internal/v1/api-keys/cleanup", {}), ("/internal/v1/api-keys/validate", {"key": "sk-oai-test-fake-key"}), ], ids=["subscriptions-select", "api-keys-cleanup", "api-keys-validate"], ) def test_internal_endpoint_not_routable(self, path, body): """POST /maas-api/internal/* through gateway must not reach the handler. Sends an authenticated request to each internal endpoint via the gateway. With the HTTPRoute scoped to /maas-api/v1, these paths have no matching route and the gateway returns a non-success status (typically 404). The response must not contain handler output. """ > _wait_for_gateway_auth_enforced() test/maas-e2e/test/e2e/tests/test_negative_security.py:757: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_helper.py:1222: in _wait_for_gateway_auth_enforced cr = _get_cr("authpolicy", name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'authpolicy', name = 'maas-gateway-auth', namespace = 'openshift-ingress' def _get_cr(kind, name, namespace=None): """Get a CR as dict, or None if not found. Retries on transient errors. Returns None only when the resource genuinely does not exist (server NotFound). Raises RuntimeError for other failures (RBAC, missing CRD, transport errors that persist after retries) so callers can distinguish infrastructure issues from true absence. """ namespace = namespace or _ns() max_retries = 3 retry_delay = 2 for attempt in range(max_retries): result = subprocess.run(["oc", "get", kind, name, "-n", namespace, "-o", "json"], capture_output=True, text=True) if result.returncode == 0: return json.loads(result.stdout) if attempt < max_retries - 1 and _is_transient_kubectl_error(result.stderr): log.warning( f"Transient kubectl error getting {kind}/{name} (attempt {attempt + 1}/{max_retries}): {result.stderr.strip()}" ) time.sleep(retry_delay * (attempt + 1)) continue # Terminal failure — distinguish not-found from other errors if _is_not_found_error(result.stderr): return None log.error( f"Failed to get {kind}/{name} in namespace '{namespace}' after {attempt + 1} attempts. " f"Last error: {result.stderr.strip()}" ) > raise RuntimeError( f"Failed to get {kind}/{name} in namespace '{namespace}': {result.stderr.strip()}" ) E RuntimeError: Failed to get authpolicy/maas-gateway-auth in namespace 'openshift-ingress': Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/test_helper.py:688: RuntimeError ------------------------------ Captured log call ------------------------------- INFO test_helper:test_helper.py:1214 Waiting for gateway AuthPolicy openshift-ingress/maas-gateway-auth Accepted+Enforced (timeout: 240s)... WARNING test_helper:test_helper.py:674 Transient kubectl error getting authpolicy/maas-gateway-auth (attempt 1/3): Unable to connect to the server: net/http: TLS handshake timeout WARNING test_helper:test_helper.py:674 Transient kubectl error getting authpolicy/maas-gateway-auth (attempt 2/3): Unable to connect to the server: net/http: TLS handshake timeout ERROR test_helper:test_helper.py:684 Failed to get authpolicy/maas-gateway-auth in namespace 'openshift-ingress' after 3 attempts. Last error: Unable to connect to the server: net/http: TLS handshake timeout ____ TestAITenantLifecycle.test_aitenant_migrates_and_removes_legacy_tenant ____ [gw1] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python self = def test_aitenant_migrates_and_removes_legacy_tenant(self): > if not _crd_exists(LEGACY_TENANT_CRD): test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:427: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ crd = 'tenants.maas.opendatahub.io' def _crd_exists(crd): result = _oc_run(["get", "crd", crd]) if result.returncode == 0: return True if _oc_output_not_found(result): return False > pytest.fail(f"`oc get crd {crd}` failed: {result.stderr.strip() or result.stdout.strip()}") E Failed: `oc get crd tenants.maas.opendatahub.io` failed: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:148: Failed _ TestInternalEndpointIsolation.test_internal_endpoint_not_routable[api-keys-validate] _ [gw4] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python self = path = '/internal/v1/api-keys/validate', body = {'key': 'sk-oai-test-fake-key'} @pytest.mark.parametrize( "path,body", [ ( "/internal/v1/subscriptions/select", {"groups": ["system:authenticated"], "username": "test"}, ), ("/internal/v1/api-keys/cleanup", {}), ("/internal/v1/api-keys/validate", {"key": "sk-oai-test-fake-key"}), ], ids=["subscriptions-select", "api-keys-cleanup", "api-keys-validate"], ) def test_internal_endpoint_not_routable(self, path, body): """POST /maas-api/internal/* through gateway must not reach the handler. Sends an authenticated request to each internal endpoint via the gateway. With the HTTPRoute scoped to /maas-api/v1, these paths have no matching route and the gateway returns a non-success status (typically 404). The response must not contain handler output. """ > _wait_for_gateway_auth_enforced() test/maas-e2e/test/e2e/tests/test_negative_security.py:757: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_helper.py:1222: in _wait_for_gateway_auth_enforced cr = _get_cr("authpolicy", name, namespace) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ kind = 'authpolicy', name = 'maas-gateway-auth', namespace = 'openshift-ingress' def _get_cr(kind, name, namespace=None): """Get a CR as dict, or None if not found. Retries on transient errors. Returns None only when the resource genuinely does not exist (server NotFound). Raises RuntimeError for other failures (RBAC, missing CRD, transport errors that persist after retries) so callers can distinguish infrastructure issues from true absence. """ namespace = namespace or _ns() max_retries = 3 retry_delay = 2 for attempt in range(max_retries): result = subprocess.run(["oc", "get", kind, name, "-n", namespace, "-o", "json"], capture_output=True, text=True) if result.returncode == 0: return json.loads(result.stdout) if attempt < max_retries - 1 and _is_transient_kubectl_error(result.stderr): log.warning( f"Transient kubectl error getting {kind}/{name} (attempt {attempt + 1}/{max_retries}): {result.stderr.strip()}" ) time.sleep(retry_delay * (attempt + 1)) continue # Terminal failure — distinguish not-found from other errors if _is_not_found_error(result.stderr): return None log.error( f"Failed to get {kind}/{name} in namespace '{namespace}' after {attempt + 1} attempts. " f"Last error: {result.stderr.strip()}" ) > raise RuntimeError( f"Failed to get {kind}/{name} in namespace '{namespace}': {result.stderr.strip()}" ) E RuntimeError: Failed to get authpolicy/maas-gateway-auth in namespace 'openshift-ingress': Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/test_helper.py:688: RuntimeError ------------------------------ Captured log call ------------------------------- INFO test_helper:test_helper.py:1214 Waiting for gateway AuthPolicy openshift-ingress/maas-gateway-auth Accepted+Enforced (timeout: 240s)... WARNING test_helper:test_helper.py:674 Transient kubectl error getting authpolicy/maas-gateway-auth (attempt 1/3): Unable to connect to the server: net/http: TLS handshake timeout WARNING test_helper:test_helper.py:674 Transient kubectl error getting authpolicy/maas-gateway-auth (attempt 2/3): Unable to connect to the server: net/http: TLS handshake timeout ERROR test_helper:test_helper.py:684 Failed to get authpolicy/maas-gateway-auth in namespace 'openshift-ingress' after 3 attempts. Last error: Unable to connect to the server: net/http: TLS handshake timeout _ TestAITenantLifecycle.test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects _ [gw1] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python self = def test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects(self): case = _new_aitenant_case() try: > _apply_gateway_fixture(case) test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:540: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:183: in _apply_gateway_fixture apply_https_gateway_fixture( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:548: in apply_gateway_fixture apply_gateway_access_label(INFRA_NAMESPACE, gateway_name) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:468: in apply_gateway_access_label ensure_namespace(namespace, labels={gateway_access_label_key(gateway_name): "true"}) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'odh-ai-gateway-infra' def ensure_namespace(name: str, *, labels: Optional[dict[str, str]] = None) -> None: result = _oc_run(["create", "namespace", name]) if result.returncode != 0 and "AlreadyExists" not in (result.stderr or "") and "already exists" not in (result.stderr or "").lower(): > raise RuntimeError(f"failed to create namespace {name}: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: failed to create namespace odh-ai-gateway-infra: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:481: RuntimeError ----------------------------- Captured stdout call ----------------------------- [cleanup] failed to delete aitenant/e2e-ait-f73f055c: `oc delete aitenant e2e-ait-f73f055c --ignore-not-found --timeout=180s -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete gateway/e2e-ait-f73f055c: `oc delete gateway e2e-ait-f73f055c --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete configmap/e2e-ait-f73f055c-gw-options: `oc delete configmap e2e-ait-f73f055c-gw-options --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: http2: client connection lost [cleanup] failed to remove gateway access label for e2e-ait-f73f055c: failed to remove gateway access label from odh-ai-gateway-infra: Unable to connect to the server: net/http: TLS handshake timeout ___ TestAITenantLifecycle.test_aitenant_derives_non_default_tenant_namespace ___ [gw1] linux -- Python 3.9.25 /workspace/source/test/e2e/.venv/bin/python self = def test_aitenant_derives_non_default_tenant_namespace(self): """RHOAIENG-66836: non-default AITenant must not use models-as-a-service tenant namespace.""" suffix = uuid.uuid4().hex[:8] aitenant_name = f"e2e-derive-{suffix}" reserved_ns = _ns() expected_ns = f"ai-tenant-{aitenant_name}" gateway_name = aitenant_name try: > _apply_gateway_fixture({"gateway_name": gateway_name, "aitenant_name": aitenant_name}) test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:601: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py:183: in _apply_gateway_fixture apply_https_gateway_fixture( test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:548: in apply_gateway_fixture apply_gateway_access_label(INFRA_NAMESPACE, gateway_name) test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:468: in apply_gateway_access_label ensure_namespace(namespace, labels={gateway_access_label_key(gateway_name): "true"}) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ name = 'odh-ai-gateway-infra' def ensure_namespace(name: str, *, labels: Optional[dict[str, str]] = None) -> None: result = _oc_run(["create", "namespace", name]) if result.returncode != 0 and "AlreadyExists" not in (result.stderr or "") and "already exists" not in (result.stderr or "").lower(): > raise RuntimeError(f"failed to create namespace {name}: {result.stderr.strip() or result.stdout.strip()}") E RuntimeError: failed to create namespace odh-ai-gateway-infra: Unable to connect to the server: net/http: TLS handshake timeout test/maas-e2e/test/e2e/tests/multitenancy_helpers.py:481: RuntimeError ----------------------------- Captured stdout call ----------------------------- [cleanup] failed to delete aitenant/e2e-derive-2665a3c0: `oc delete aitenant e2e-derive-2665a3c0 --ignore-not-found --timeout=180s -n ai-tenants` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete gateway/e2e-derive-2665a3c0: `oc delete gateway e2e-derive-2665a3c0 --ignore-not-found --timeout=60s -n openshift-ingress` failed: Unable to connect to the server: net/http: TLS handshake timeout [cleanup] failed to delete configmap/e2e-derive-2665a3c0-gw-options: Command '['/usr/local/bin/oc', 'delete', 'configmap', 'e2e-derive-2665a3c0-gw-options', '--ignore-not-found', '--timeout=60s', '-n', 'openshift-ingress']' timed out after 90 seconds [cleanup] failed to remove gateway access label for e2e-derive-2665a3c0: failed to remove gateway access label from odh-ai-gateway-infra: Unable to connect to the server: net/http: TLS handshake timeout ------ generated xml file: /workspace/artifacts-dir/e2e-system:admin.xml ------- - Generated html report: file:///workspace/artifacts-dir/e2e-system%3Aadmin.html - =========================== short test summary info ============================ FAILED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestWebhookValidation::test_subscription_rejected_in_unlabeled_namespace@security FAILED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestWebhookValidation::test_authpolicy_rejected_in_unlabeled_namespace@security FAILED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[subscriptions-select]@security FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_create_bootstrap_resources@mt_lifecycle FAILED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-cleanup]@security FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_migrates_and_removes_legacy_tenant@mt_lifecycle FAILED test/maas-e2e/test/e2e/tests/test_negative_security.py::TestInternalEndpointIsolation::test_internal_endpoint_not_routable[api-keys-validate]@security FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_delete_cleans_maas_resources_and_preserves_user_objects@mt_lifecycle FAILED test/maas-e2e/test/e2e/tests/test_aitenant_lifecycle.py::TestAITenantLifecycle::test_aitenant_derives_non_default_tenant_namespace@mt_lifecycle ERROR test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_create_api_key@api_keys ERROR test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_explicit_subscription_header@models ERROR test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_list_api_keys@api_keys ERROR test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_empty_subscription_header_value@models ERROR test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyCRUD::test_revoke_api_key@api_keys ERROR test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_models_filtered_by_subscription@models ERROR test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_admin_manage_other_users_keys@api_keys ERROR test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_deduplication_same_model_multiple_refs@models ERROR test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeyAuthorization::test_non_admin_cannot_access_other_users_keys@api_keys ERROR test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_multiple_distinct_models_in_subscription@models ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_model_routes_through_tenant_gateway@tenant_isolation ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_inference_succeeds_through_tenant_gateway@tenant_isolation ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantModelInference::test_tenant_isolation_cross_gateway_blocked@tenant_isolation ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_correct_model_in_body_succeeds@tenant_isolation ERROR test/maas-e2e/test/e2e/tests/test_tenant_model_inference.py::TestTenantBodyRouting::test_wrong_model_in_body_rejected@tenant_isolation ERROR test/maas-e2e/test/e2e/tests/test_tenant_namespace_discovery.py::TestTenantNamespaceDiscovery::test_labeled_tenant_namespace_is_discovered@mt_lifecycle !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 25 failures !!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! = 9 failed, 31 passed, 8 skipped, 15 warnings, 16 errors in 476.99s (0:07:56) == Running E2E pass 2/2: serial cluster mutators (-m serial, single worker) ============================= test session starts ============================== platform linux -- Python 3.9.25, pytest-8.4.2, pluggy-1.6.0 -- /workspace/source/test/e2e/.venv/bin/python cachedir: .pytest_cache metadata: {'Python': '3.9.25', 'Platform': 'Linux-5.14.0-570.134.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '8.4.2', 'pluggy': '1.6.0'}, 'Plugins': {'xdist': '3.8.0', 'metadata': '3.1.1', 'html': '4.2.0'}, 'PLATFORM': 'el9'} rootdir: /workspace/source/test/maas-e2e/test/e2e configfile: pyproject.toml plugins: xdist-3.8.0, metadata-3.1.1, html-4.2.0 collecting ... collected 234 items / 211 deselected / 23 selected test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_pending_subscription [token] using env TOKEN (masked): 1016 FAILED [ 4%] test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_reject_key_for_unreconciled_subscription PASSED [ 8%] test/maas-e2e/test/e2e/tests/test_negative_security.py::TestAuthPolicyRemoval::test_authpolicy_deletion_revokes_access PASSED [ 13%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_rate_limit_exhaustion_gets_429 PASSED [ 17%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestSubscriptionEnforcement::test_models_endpoint_exempt_from_rate_limiting PASSED [ 21%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestMultipleSubscriptionsPerModel::test_user_in_one_of_two_subscriptions_gets_200 PASSED [ 26%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestMultipleAuthPoliciesPerModel::test_delete_one_auth_policy_other_still_works PASSED [ 30%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestCascadeDeletion::test_delete_subscription_rebuilds_trlp PASSED [ 34%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestCascadeDeletion::test_trlp_persists_during_multi_subscription_deletion PASSED [ 39%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestCascadeDeletion::test_delete_last_subscription_denies_access PASSED [ 43%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestCascadeDeletion::test_unconfigured_model_denied_by_gateway_auth PASSED [ 47%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_with_access_but_no_subscription_gets_403 PASSED [ 52%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_single_subscription_auto_selects PASSED [ 56%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestE2ESubscriptionFlow::test_e2e_group_based_auth_but_no_subscription_gets_403 PASSED [ 60%] test/maas-e2e/test/e2e/tests/test_subscription.py::TestStatusReporting::test_subscription_degraded_trlp_blocks_inference PASSED [ 65%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_single_subscription_auto_select PASSED [ 69%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_different_modelrefs_same_model_id maasauthpolicy.maas.opendatahub.io/e2e-diff-refs-auth created maassubscription.maas.opendatahub.io/e2e-diff-refs-subscription created PASSED [ 73%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_empty_model_list PASSED [ 78%] test/maas-e2e/test/e2e/tests/test_models_endpoint.py::TestModelsEndpoint::test_central_models_endpoint_exempt_from_rate_limiting PASSED [ 82%] test/maas-e2e/test/e2e/tests/test_tenant.py::TestTenantLifecycle::test_payload_processing_deployed_with_active_tenant PASSED [ 86%] test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingGovernance::test_embedding_default_deny_403 PASSED [ 91%] test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingGovernance::test_embedding_trlp_429 PASSED [ 95%] test/maas-e2e/test/e2e/tests/test_embedding_inference.py::TestEmbeddingGovernance::test_embedding_with_governance_200 PASSED [100%] =================================== FAILURES =================================== ____ TestAPIKeySubscriptionPhases.test_create_key_for_pending_subscription _____ self = @pytest.mark.serial def test_create_key_for_pending_subscription(self): """API key creation succeeds for Pending subscription.""" ns = _ns() subscription_name = "e2e-apikey-pending-sub" auth_name = "e2e-apikey-pending-auth" sa_name = "e2e-apikey-pending-sa" try: oc_token = _create_sa_token(sa_name, namespace=MODEL_NAMESPACE) sa_user = _sa_to_user(sa_name, namespace=MODEL_NAMESPACE) > _create_test_auth_policy(auth_name, MODEL_REF, users=[sa_user]) test/maas-e2e/test/e2e/tests/test_api_keys.py:1663: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test/maas-e2e/test/e2e/tests/test_helper.py:916: in _create_test_auth_policy _apply_cr({ test/maas-e2e/test/e2e/tests/test_helper.py:620: in _apply_cr subprocess.run(["oc", "apply", "-f", "-"], input=json.dumps(cr_dict), capture_output=True, text=True, check=True) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ input = '{"apiVersion": "maas.opendatahub.io/v1alpha1", "kind": "MaaSAuthPolicy", "metadata": {"name": "e2e-apikey-pending-aut...ted", "namespace": "llm"}], "subjects": {"users": ["system:serviceaccount:llm:e2e-apikey-pending-sa"], "groups": []}}}' capture_output = True, timeout = None, check = True popenargs = (['oc', 'apply', '-f', '-'],) kwargs = {'stderr': -1, 'stdin': -1, 'stdout': -1, 'text': True} process = , stdout = '' stderr = 'Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vmaa...3/validate-maas-opendatahub-io-v1alpha1-maasauthpolicy?timeout=10s": http: server gave HTTP response to HTTPS client\n' retcode = 1 def run(*popenargs, input=None, capture_output=False, timeout=None, check=False, **kwargs): """Run command with arguments and return a CompletedProcess instance. The returned instance will have attributes args, returncode, stdout and stderr. By default, stdout and stderr are not captured, and those attributes will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them. If check is True and the exit code was non-zero, it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute, and output & stderr attributes if those streams were captured. If timeout is given, and the process takes too long, a TimeoutExpired exception will be raised. There is an optional argument "input", allowing you to pass bytes or a string to the subprocess's stdin. If you use this argument you may not also use the Popen constructor's "stdin" argument, as it will be used internally. By default, all communication is in bytes, and therefore any "input" should be bytes, and the stdout and stderr will be bytes. If in text mode, any "input" should be a string, and stdout and stderr will be strings decoded according to locale encoding, or by "encoding" if set. Text mode is triggered by setting any of text, encoding, errors or universal_newlines. The other arguments are the same as for the Popen constructor. """ if input is not None: if kwargs.get('stdin') is not None: raise ValueError('stdin and input arguments may not both be used.') kwargs['stdin'] = PIPE if capture_output: if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None: raise ValueError('stdout and stderr arguments may not be used ' 'with capture_output.') kwargs['stdout'] = PIPE kwargs['stderr'] = PIPE with Popen(*popenargs, **kwargs) as process: try: stdout, stderr = process.communicate(input, timeout=timeout) except TimeoutExpired as exc: process.kill() if _mswindows: # Windows accumulates the output in a single blocking # read() call run on child threads, with the timeout # being done in a join() on those threads. communicate() # _after_ kill() is required to collect that and add it # to the exception. exc.stdout, exc.stderr = process.communicate() else: # POSIX _communicate already populated the output so # far into the TimeoutExpired exception. process.wait() raise except: # Including KeyboardInterrupt, communicate handled that. process.kill() # We don't call process.wait() as .__exit__ does that for us. raise retcode = process.poll() if check and retcode: > raise CalledProcessError(retcode, process.args, output=stdout, stderr=stderr) E subprocess.CalledProcessError: Command '['oc', 'apply', '-f', '-']' returned non-zero exit status 1. /usr/lib64/python3.9/subprocess.py:528: CalledProcessError ---------------------------- Captured stdout setup ----------------------------- [token] using env TOKEN (masked): 1016 ------------------------------ Captured log setup ------------------------------ INFO test_helper:test_helper.py:1214 Waiting for gateway AuthPolicy openshift-ingress/maas-gateway-auth Accepted+Enforced (timeout: 240s)... INFO test_helper:test_helper.py:1254 Gateway AuthPolicy openshift-ingress/maas-gateway-auth is Accepted and Enforced INFO test_api_keys:test_api_keys.py:220 Gateway warm-up completed: HTTP 201 ------------------------------ Captured log call ------------------------------- INFO test_helper:test_helper.py:915 Creating MaaSAuthPolicy: e2e-apikey-pending-auth INFO test_helper:test_helper.py:1612 Scaling maas-controller to 1 replicas in namespace opendatahub... INFO test_helper:test_helper.py:1649 ✓ maas-controller scaled to 1 replica(s) --- generated xml file: /workspace/artifacts-dir/e2e-system:admin-serial.xml --- - Generated html report: file:///workspace/artifacts-dir/e2e-system%3Aadmin-serial.html - =========================== short test summary info ============================ FAILED test/maas-e2e/test/e2e/tests/test_api_keys.py::TestAPIKeySubscriptionPhases::test_create_key_for_pending_subscription ==== 1 failed, 22 passed, 211 deselected, 26 warnings in 494.74s (0:08:14) ===== ERROR: E2E tests failed (parallel_rc=1, serial_rc=1)