2026-10-30T18:42:06.926840Z info FLAG: --concurrency="0" 2026-10-30T18:42:06.926865Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T18:42:06.926870Z info FLAG: --help="false" 2026-10-30T18:42:06.926874Z info FLAG: --log_as_json="false" 2026-10-30T18:42:06.926877Z info FLAG: --log_caller="" 2026-10-30T18:42:06.926880Z info FLAG: --log_output_level="default:info" 2026-10-30T18:42:06.926883Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T18:42:06.926891Z info FLAG: --log_target="[stdout]" 2026-10-30T18:42:06.926894Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T18:42:06.926897Z info FLAG: --outlierLogPath="" 2026-10-30T18:42:06.926900Z info FLAG: --profiling="true" 2026-10-30T18:42:06.926903Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T18:42:06.926906Z info FLAG: --proxyLogLevel="warning" 2026-10-30T18:42:06.926910Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T18:42:06.926913Z info FLAG: --stsPort="0" 2026-10-30T18:42:06.926916Z info FLAG: --templateFile="" 2026-10-30T18:42:06.926919Z info FLAG: --tokenManagerPlugin="" 2026-10-30T18:42:06.926924Z info FLAG: --vklog="0" 2026-10-30T18:42:06.926928Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T18:42:06.927145Z info Proxy role ips=[10.132.0.57] type=router id=maas-default-gateway-openshift-default-855f9574cf-p8smm.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T18:42:06.927206Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T18:42:06.928867Z info cpu limit detected as 2, setting concurrency 2026-10-30T18:42:06.929197Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T18:42:06.929220Z info JWT policy is third-party-jwt 2026-10-30T18:42:06.929224Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T18:42:06.931683Z info Starting default Istio SDS Server 2026-10-30T18:42:06.932156Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T18:42:06.932159Z info Opening status port 15020 2026-10-30T18:42:06.932222Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T18:42:06.933344Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T18:42:06.935000Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T18:42:06.935983Z info Starting proxy agent 2026-10-30T18:42:06.936023Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T18:42:06.946127Z info sds Starting SDS grpc server 2026-10-30T18:42:06.946148Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T18:42:06.997932Z info cache generated new workload certificate resourceName=default latency=64.467174ms ttl=23h59m59.00207328s 2026-10-30T18:42:06.997977Z info cache Root cert has changed, start rotating root cert 2026-10-30T18:42:06.998018Z info cache returned workload trust anchor from cache ttl=23h59m59.001982736s 2026-10-30T18:42:07.009859Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T18:42:07.135939Z info ads ADS: new connection for node:1 2026-10-30T18:42:07.136008Z info cache returned workload certificate from cache ttl=23h59m58.863993232s 2026-10-30T18:42:07.136492Z info ads ADS: new connection for node:2 2026-10-30T18:42:07.136581Z info cache returned workload trust anchor from cache ttl=23h59m58.863420242s 2026-10-30T18:42:07.218142Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T18:42:15.247526Z info Readiness succeeded in 8.323325337s 2026-10-30T18:42:15.247834Z info Envoy proxy is ready 2026-10-30T18:45:27.570999Z info ads ADS: new connection for node:3 2026-10-30T18:45:27.578050Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T18:45:27.578491Z info cache read certificate from file resource=file-root:system 2026-10-30T18:45:28.167942Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T18:45:35.359724Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T18:45:44.801166Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T18:46:02.960847Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T18:46:29.078871Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T18:46:29.078905Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T18:46:29.078916Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T18:46:29.078920Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T18:46:29.079147Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T18:46:28.873Z] "POST /anything/e2e-rt-eq-1e8613-client HTTP/1.1" 200 - via_upstream - "-" 124 1776 206 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "eceac68a-207a-4006-bdc2-62d521b08d7c" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.57:50232 10.132.0.57:443 10.132.0.19:60206 - llm.external-model-e2e-rt-eq-1e8613-4d38146b8aab.2 2026-10-30T18:46:31.706142Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:46:31.706189Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:46:31.706205Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:46:31.706211Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T18:46:31.706477Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T18:46:31.280Z] "POST /anything/e2e-rt-eq-1e8613-client HTTP/1.1" 200 - via_upstream - "-" 124 1777 425 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "76036d79-a252-4625-9a59-0f9ece8ac3e3" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.57:50246 10.132.0.57:443 10.132.0.19:60232 - llm.external-model-e2e-rt-eq-1e8613-4d38146b8aab.2 2026-10-30T18:46:33.617706Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:46:33.617846Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:46:33.617896Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:46:33.617926Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T18:46:33.618283Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T18:46:33.429Z] "POST /anything/e2e-rt-eq-1e8613-client HTTP/1.1" 200 - via_upstream - "-" 124 1768 188 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "36a2f7aa-cfb4-4ed4-8fcf-b2cd1d7f55b6" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.57:50260 10.132.0.57:443 10.132.0.19:60246 - llm.external-model-e2e-rt-eq-1e8613-4d38146b8aab.2 2026-10-30T18:46:57.419613Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T18:47:02.505892Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T18:47:02.692248Z info ads ADS: "" 3 terminated 2026-10-30T18:47:14.233151Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T18:50:38.947Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 40 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "cd2efa82-166c-4aa2-8acc-628b809a5c00" "maas.apps.3ad58fbeb6df127dd4c4.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.132.0.57:443 10.132.0.19:52846 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T18:50:41.395Z] "POST /maas-api/v1/api-keys HTTP/1.1" 403 - - "-" 77 0 43 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "602a0a42-6919-4f44-9ee0-443aae45b3d1" "maas.apps.3ad58fbeb6df127dd4c4.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.132.0.57:443 10.132.0.19:52854 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T18:50:58.123Z] "POST /maas-api/internal/v1/api-keys/cleanup HTTP/1.1" 404 NR route_not_found - "-" 2 0 16 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "fa46f95d-711a-419c-9644-6966c30c3a70" "maas.apps.3ad58fbeb6df127dd4c4.prod.konflux-ocp-ci.dev" "-" - - 10.132.0.57:443 10.132.0.19:44798 - - 2026-10-30T18:52:59.795019Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T18:53:59.824157Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T18:53:59.824193Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T18:53:59.824207Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T18:53:59.824211Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-10-30T18:53:59.723Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 69 311 101 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "b5686f5e-df53-41b8-aee6-3a0d392cddb1" "maas.apps.3ad58fbeb6df127dd4c4.prod.konflux-ocp-ci.dev" "10.133.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.57:45246 10.132.0.57:443 10.132.0.19:42788 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T18:55:02.937463Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T18:55:43.067506Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:55:43.067541Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:55:43.067554Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T18:55:43.067558Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T18:55:43.067720Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 2026-10-30T18:55:43.067772Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T18:55:43.067780Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T18:55:43.067783Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T18:55:43.067786Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-30T18:55:42.985Z] "POST /v1/completions HTTP/1.1" 404 - via_upstream - "-" 94 126 82 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "964d466a-c17a-4f74-a7bd-c61e65fd9aee" "maas.apps.3ad58fbeb6df127dd4c4.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.57:32792 10.132.0.57:443 10.132.0.19:37482 - v1-completions-path 2026-10-30T18:58:42.206495Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: gRPC status code is not OK thread=24 [2026-10-30T18:58:40.181Z] "POST /maas-api/v1/api-keys HTTP/1.1" 500 - - "-" 72 23 2025 - "52.71.36.72,10.132.0.19" "python-requests/2.34.2" "514075b0-9091-4be2-994c-2d4faa5bbd66" "maas.apps.3ad58fbeb6df127dd4c4.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.132.0.57:443 10.132.0.19:53622 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T19:05:19.820038Z info ads ADS: new connection for node:4 2026-10-30T19:05:19.851767Z info cache read certificate from file resource=file-root:system 2026-10-30T19:13:31.472789Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T19:13:48.262002Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T19:14:26.235996Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T19:14:53.797035Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-30T19:18:59.397151Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T19:19:01.394755Z info ads ADS: new connection for node:5 2026-10-30T19:19:01.395054Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T19:19:01.395177Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T19:19:04.621103Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T19:19:07.858537Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T19:19:09.530316Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T19:19:11.065210Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T19:19:34.495228Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T19:21:39.875553Z info ads ADS: "" 4 terminated 2026-10-30T19:22:29.323388Z info ads ADS: "" 5 terminated with stream closed 2026-10-30T19:25:21.321711Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3