Fetching https://github.com/opendatahub-io/models-as-a-service@7879675aa66d002425c7b32375d6d0156eaa0c56 into /workspace/source/test/maas-e2e ... MaaS checkout at: 7879675aa66d002425c7b32375d6d0156eaa0c56 Patched /workspace/source/test/maas-e2e/scripts/deploy.sh Patched /workspace/source/test/maas-e2e/test/e2e/scripts/deploy-models.sh (MaaS checkout fixture root) NOTE: rewriting AI_GATEWAY_CONTROLLER_IMAGE from snapshot digest to tag odh-stable MaaS platform images: MAAS_API_IMAGE=quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 MAAS_CONTROLLER_IMAGE=quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 AI_GATEWAY_CONTROLLER_IMAGE=quay.io/opendatahub/odh-ai-gateway-controller:odh-stable PRAXIS_EXTPROC_IMAGE=quay.io/opendatahub/odh-praxis-extproc:odh-stable ---------------------------------------- ai-gateway-controller E2E on OpenShift ---------------------------------------- Checking prerequisites... Prerequisites met — logged in as: system:admin DEPLOY_MODE: kustomize MAAS_API_IMAGE: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 MAAS_CONTROLLER_IMAGE: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 AI_GATEWAY_CONTROLLER_IMAGE: quay.io/opendatahub/odh-ai-gateway-controller:odh-stable PRAXIS_EXTPROC_IMAGE: quay.io/opendatahub/odh-praxis-extproc:odh-stable 2026-10-30T12:51:03Z deploy_platform start Deploying MaaS platform via ODH operator... Gateway ingress mode for deploy.sh: ocproute (loadbalancer | ocproute) Using custom MaaS API image: quay.io/opendatahub/maas-api:7879675aa66d002425c7b32375d6d0156eaa0c56 Using custom MaaS controller image: quay.io/opendatahub/maas-controller:7879675aa66d002425c7b32375d6d0156eaa0c56 Deployment mode: kustomize Installing OpenDataHub operator... ========================================= 🚀 OpenDataHub (ODH) Installation ========================================= 1️⃣ Installing ODH Operator from repository manifests... Using operator image: quay.io/opendatahub/opendatahub-operator:latest namespace/opendatahub created /tmp/tmp.qu5DZKJAkd /workspace/source /tmp/tmp.qu5DZKJAkd/opendatahub-operator /tmp/tmp.qu5DZKJAkd /workspace/source go: go.mod requires go >= 1.26.5 (running go 1.25.5; GOTOOLCHAIN=local) mkdir -p /tmp/tmp.qu5DZKJAkd/opendatahub-operator/bin Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.17.3 go: downloading go1.26.5 (linux/amd64) go: downloading sigs.k8s.io/controller-tools v0.17.3 go: downloading golang.org/x/tools v0.30.0 go: downloading github.com/spf13/cobra v1.9.1 go: downloading github.com/fatih/color v1.18.0 go: downloading k8s.io/apimachinery v0.32.2 go: downloading k8s.io/apiextensions-apiserver v0.32.2 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading k8s.io/api v0.32.2 go: downloading k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738 go: downloading github.com/gobuffalo/flect v1.0.3 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading golang.org/x/sys v0.30.0 go: downloading github.com/spf13/pflag v1.0.6 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading k8s.io/klog/v2 v2.130.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.2 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 go: downloading golang.org/x/net v0.35.0 go: downloading github.com/fxamacker/cbor/v2 v2.7.0 go: downloading github.com/go-logr/logr v1.4.2 go: downloading github.com/x448/float16 v0.8.4 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading golang.org/x/sync v0.11.0 go: downloading golang.org/x/mod v0.23.0 go: downloading golang.org/x/text v0.22.0 Downloading sigs.k8s.io/kustomize/kustomize/v5@v5.8.1 go: downloading sigs.k8s.io/kustomize/kustomize/v5 v5.8.1 go: downloading sigs.k8s.io/kustomize/api v0.21.1 go: downloading sigs.k8s.io/kustomize/cmd/config v0.21.1 go: downloading sigs.k8s.io/yaml v1.5.0 go: downloading sigs.k8s.io/kustomize/kyaml v0.21.1 go: downloading golang.org/x/text v0.28.0 go: downloading go.yaml.in/yaml/v2 v2.4.2 go: downloading github.com/go-errors/errors v1.4.2 go: downloading github.com/blang/semver/v4 v4.0.0 go: downloading gopkg.in/evanphx/json-patch.v4 v4.13.0 go: downloading k8s.io/kube-openapi v0.0.0-20241212222426-2c72e554b1e7 go: downloading github.com/davecgh/go-spew v1.1.1 go: downloading go.yaml.in/yaml/v3 v3.0.3 go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 go: downloading github.com/xlab/treeprint v1.2.0 go: downloading github.com/sergi/go-diff v1.4.0 go: downloading github.com/google/gnostic-models v0.6.9 go: downloading google.golang.org/protobuf v1.36.5 go: downloading github.com/go-openapi/swag v0.23.0 go: downloading github.com/go-openapi/jsonreference v0.20.2 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/go-openapi/jsonpointer v0.21.0 go: downloading github.com/josharian/intern v1.0.0 Downloading github.com/mikefarah/yq/v4@v4.53.2 go: downloading github.com/mikefarah/yq/v4 v4.53.2 go: downloading github.com/yuin/gopher-lua v1.1.2 go: downloading github.com/alecthomas/participle/v2 v2.1.4 go: downloading github.com/spf13/pflag v1.0.10 go: downloading github.com/zclconf/go-cty v1.18.0 go: downloading go.yaml.in/yaml/v4 v4.0.0-rc.4 go: downloading github.com/goccy/go-yaml v1.19.2 go: downloading golang.org/x/net v0.52.0 go: downloading github.com/magiconair/properties v1.8.10 go: downloading github.com/a8m/envsubst v1.4.3 go: downloading github.com/dimchansky/utfbom v1.1.1 go: downloading github.com/go-ini/ini v1.67.0 go: downloading github.com/goccy/go-json v0.10.6 go: downloading golang.org/x/text v0.35.0 go: downloading github.com/fatih/color v1.19.0 go: downloading github.com/hashicorp/hcl/v2 v2.24.0 go: downloading github.com/elliotchance/orderedmap v1.8.0 go: downloading github.com/pelletier/go-toml/v2 v2.3.0 go: downloading github.com/spf13/cobra v1.10.2 go: downloading github.com/jinzhu/copier v0.4.0 go: downloading github.com/mattn/go-colorable v0.1.14 go: downloading golang.org/x/sys v0.42.0 go: downloading github.com/apparentlymart/go-textseg/v15 v15.0.0 go: downloading github.com/agext/levenshtein v1.2.1 go: downloading github.com/google/go-cmp v0.6.0 go: downloading github.com/mitchellh/go-wordwrap v1.0.1 go: go.mod requires go >= 1.26.5 (running go 1.25.5; GOTOOLCHAIN=local) go: go.mod requires go >= 1.26.5 (running go 1.25.5; GOTOOLCHAIN=local) go: go.mod requires go >= 1.26.5 (running go 1.25.5; GOTOOLCHAIN=local) go: go.mod requires go >= 1.26.5 (running go 1.25.5; GOTOOLCHAIN=local) go: go.mod requires go >= 1.26.5 (running go 1.25.5; GOTOOLCHAIN=local) go: go.mod requires go >= 1.26.5 (running go 1.25.5; GOTOOLCHAIN=local) Error: load packages in root "/tmp/tmp.qu5DZKJAkd/opendatahub-operator": err: exit status 1: stderr: go: go.mod requires go >= 1.26.5 (running go 1.25.5; GOTOOLCHAIN=local) Usage: controller-gen [flags] Examples: # Generate RBAC manifests and crds for all types under apis/, # outputting crds to /tmp/crds and everything else to stdout controller-gen rbac:roleName= crd paths=./apis/... output:crd:dir=/tmp/crds output:stdout # Generate deepcopy/runtime.Object implementations for a particular file controller-gen object paths=./apis/v1beta1/some_types.go # Generate OpenAPI v3 schemas for API packages and merge them into existing CRD manifests controller-gen schemapatch:manifests=./manifests output:dir=./manifests paths=./pkg/apis/... # Run all the generators for a given project controller-gen paths=./apis/... # Explain the markers for generating CRDs, and their arguments controller-gen crd -ww Flags: -h, --detailed-help count print out more detailed help (up to -hhh for the most detailed output, or -hhhh for json output) --help print out usage and a summary of options --load-build-tags strings build tags to use when loading Go packages (default [ignore_autogenerated]) --version show version -w, --which-markers count print out all markers available with the requested generators (up to -www for the most detailed output, or -wwww for json output) Options generators +webhook[:headerFile=][,year=] package generates (partial) {Mutating,Validating}WebhookConfiguration objects. +schemapatch[:generateEmbeddedObjectMeta=],manifests=[,maxDescLen=] package patches existing CRDs with new schemata. +rbac[:headerFile=],roleName=[,year=] package generates ClusterRole objects. +object[:headerFile=][,year=] package generates code containing DeepCopy, DeepCopyInto, and +crd[:allowDangerousTypes=][,crdVersions=<[]string>][,deprecatedV1beta1CompatibilityPreserveUnknownFields=][,generateEmbeddedObjectMeta=][,headerFile=][,ignoreUnexportedFields=][,maxDescLen=][,year=] package generates CustomResourceDefinition objects. generic +paths=<[]string> package represents paths and go-style path patterns to use as package roots. output rules (optionally as output::...) +output:artifacts[:code=],config= package outputs artifacts to different locations, depending on +output:dir= package outputs each artifact to the given directory, regardless +output:none package skips outputting anything. +output:stdout package outputs everything to standard-out, with no separation. run `controller-gen --load-build-tags=odh rbac:roleName=controller-manager-role crd:ignoreUnexportedFields=true webhook paths={} output:crd:artifacts:config=config/crd/bases output:rbac:artifacts:config=config/rbac output:webhook:artifacts:config=config/webhook -w` to see all available markers, or `controller-gen --load-build-tags=odh rbac:roleName=controller-manager-role crd:ignoreUnexportedFields=true webhook paths={} output:crd:artifacts:config=config/crd/bases output:rbac:artifacts:config=config/rbac output:webhook:artifacts:config=config/webhook -h` for usage make: *** [Makefile:273: manifests] Error 1 ❌ ERROR: ODH installation failed ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs No MaaS CRs found on the cluster Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 0 0 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Skipping namespace odh-ai-gateway-infra (not found) Skipping namespace models-as-a-service (not found) Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Skipping namespace kuadrant-system (not found) Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 1 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Skipping namespace llm (not found) Skipping namespace istio-system (not found) ============================================== ========== E2E Artifact Collection ========== Artifact dir: /workspace/artifacts-dir Collecting Authorino logs (token-redacted) to /workspace/artifacts-dir/authorino-debug.log Collecting cluster state to /workspace/artifacts-dir Saved to /workspace/artifacts-dir/cluster-state.log Collecting MaaS CR definitions to /workspace/artifacts-dir/maas-crs No MaaS CRs found on the cluster Collecting pod logs from namespace opendatahub to /workspace/artifacts-dir/pod-logs/opendatahub Saved 0 0 pod log file(s) to /workspace/artifacts-dir/pod-logs/opendatahub Skipping namespace odh-ai-gateway-infra (not found) Skipping namespace models-as-a-service (not found) Skipping namespace redhat-ods-operator (not found) Skipping namespace redhat-ods-applications (not found) Skipping namespace kuadrant-system (not found) Collecting pod logs from namespace openshift-ingress to /workspace/artifacts-dir/pod-logs/openshift-ingress Saved 1 pod log file(s) to /workspace/artifacts-dir/pod-logs/openshift-ingress Skipping namespace llm (not found) Skipping namespace istio-system (not found) ============================================== Collecting AIGC cluster CR extensions to /workspace/artifacts-dir/maas-crs No AIGC cluster CR extensions found on the cluster Collecting MaaS diagnostics to /workspace/artifacts-dir/gather-maas (log: /workspace/artifacts-dir/maas-must-gather.log) ... MaaS must-gather complete (see /workspace/artifacts-dir/maas-must-gather.log) ======================================== Cluster / Namespace Info ======================================== --- Current context --- admin --- Logged-in user --- system:admin --- Cluster domain --- apps.0d287c302a7cb6a93dac.prod.konflux-ocp-ci.dev DEPLOYMENT_NAMESPACE: opendatahub MAAS_API_DEPLOYMENT_NAMESPACE: odh-ai-gateway-infra MAAS_SUBSCRIPTION_NAMESPACE: models-as-a-service AUTHORINO_NAMESPACE: kuadrant-system ======================================== MaaS API Deployment ======================================== --- maas-api pods --- --- maas-api service --- ======================================== maas-controller ======================================== --- maas-controller pods --- --- maas-controller namespace env --- namespace env=N/A ======================================== Kuadrant Policies ======================================== --- AuthPolicies (all namespaces) --- --- TokenRateLimitPolicies (all namespaces) --- ======================================== MaaS CRs (maas.opendatahub.io) ======================================== --- Configs (cluster-scoped) --- --- AITenants (all namespaces) --- --- Tenants (all namespaces) --- --- MaaSAuthPolicies --- --- MaaSSubscriptions --- --- MaaSSubscription status details --- --- MaaSModelRefs (all namespaces) --- --- MaasTenantConfigs --- --- MaasTenantConfig status details --- --- ExternalModels (maas group) --- ======================================== Inference CRs (inference.opendatahub.io) ======================================== --- ExternalModels (inference group) --- --- ExternalProviders (inference group) --- ======================================== AI Gateway CRs ======================================== --- AIGateways (cluster-scoped) --- --- LLMBatchGateways (all namespaces) --- ======================================== KServe Resources ======================================== --- LLMInferenceServices (all namespaces) --- ======================================== Kuadrant Policies (managed by maas-controller) ======================================== --- AuthPolicies (managed) --- --- TokenRateLimitPolicies (managed) --- ======================================== Payload Processing / IPP ======================================== --- IPP Deployments (openshift-ingress) --- --- IPP Services (openshift-ingress) --- --- EnvoyFilters (openshift-ingress) --- --- IPP ConfigMaps (openshift-ingress) --- ======================================== NetworkPolicies ======================================== --- NetworkPolicies (openshift-ingress) --- NAME POD-SELECTOR AGE openshift-ingress-deny-all 11m router-default ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default 11m ======================================== Test User Information ======================================== No test token available (not logged in via oc) ======================================== Subscription → Model Mapping ======================================== No subscriptions found in models-as-a-service ======================================== Available Models (MaaSModelRefs) ======================================== No MaaSModelRefs found ======================================== Gateway / HTTPRoutes ======================================== --- Gateway --- --- HTTPRoutes (maas-api) --- ======================================== Authorino ======================================== --- Authorino pods --- --- ======================================== Subscription Selector Endpoint Validation ======================================== Expected URL (from resolved maas-api deployment namespace): https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select (MAAS_API_DEPLOYMENT_NAMESPACE resolved to: odh-ai-gateway-infra) --- Sample AuthPolicy subscription-info configuration --- No managed AuthPolicies found --- Connectivity test (from openshift-ingress, simulates Authorino) --- curl -vsk -m 10 -X POST 'https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select' -H 'Content-Type: application/json' -d '{}' kubectl run failed or timed out ======================================== DNS Resolution Check ======================================== Resolving: maas-api.odh-ai-gateway-infra.svc.cluster.local nslookup failed ======================================== Configuration Summary ======================================== This summary helps compare local vs CI runs: AITenants (all namespaces): 0 MaaSModelRefs (all namespaces): MaaSSubscriptions (models-as-a-service): MaaSAuthPolicies (models-as-a-service): 0 Generated Kuadrant AuthPolicies: 0 Generated Kuadrant TokenRateLimitPolicies: 0 LLMInferenceServices (all namespaces): 0 ExternalModels - inference (all namespaces): 0 ExternalProviders - inference (all namespaces): 0 EnvoyFilters (openshift-ingress): 0 IPP Deployments (openshift-ingress): 0 NetworkPolicies (openshift-ingress): 2 Subscription selector URL: https://maas-api.odh-ai-gateway-infra.svc.cluster.local:8443/internal/v1/subscriptions/select Test user: system:admin