2026-11-02T14:37:12.442156Z info FLAG: --concurrency="0" 2026-11-02T14:37:12.442177Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T14:37:12.442182Z info FLAG: --help="false" 2026-11-02T14:37:12.442184Z info FLAG: --log_as_json="false" 2026-11-02T14:37:12.442186Z info FLAG: --log_caller="" 2026-11-02T14:37:12.442188Z info FLAG: --log_output_level="default:info" 2026-11-02T14:37:12.442190Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T14:37:12.442195Z info FLAG: --log_target="[stdout]" 2026-11-02T14:37:12.442199Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T14:37:12.442202Z info FLAG: --outlierLogPath="" 2026-11-02T14:37:12.442205Z info FLAG: --profiling="true" 2026-11-02T14:37:12.442208Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T14:37:12.442211Z info FLAG: --proxyLogLevel="warning" 2026-11-02T14:37:12.442214Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T14:37:12.442217Z info FLAG: --stsPort="0" 2026-11-02T14:37:12.442218Z info FLAG: --templateFile="" 2026-11-02T14:37:12.442220Z info FLAG: --tokenManagerPlugin="" 2026-11-02T14:37:12.442223Z info FLAG: --vklog="0" 2026-11-02T14:37:12.442227Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T14:37:12.442392Z info Proxy role ips=[10.133.0.236] type=router id=maas-default-gateway-openshift-default-55fbd65575-t9gjp.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T14:37:12.442454Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T14:37:12.443688Z info cpu limit detected as 2, setting concurrency 2026-11-02T14:37:12.443929Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T14:37:12.443941Z info JWT policy is third-party-jwt 2026-11-02T14:37:12.443943Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T14:37:12.446925Z info Starting default Istio SDS Server 2026-11-02T14:37:12.446945Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T14:37:12.446967Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T14:37:12.447055Z info Opening status port 15020 2026-11-02T14:37:12.447566Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T14:37:12.448992Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T14:37:12.449806Z info Starting proxy agent 2026-11-02T14:37:12.449836Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T14:37:12.467014Z info sds Starting SDS grpc server 2026-11-02T14:37:12.467046Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T14:37:12.518008Z info cache generated new workload certificate resourceName=default latency=69.969324ms ttl=23h59m59.481996348s 2026-11-02T14:37:12.518051Z info cache Root cert has changed, start rotating root cert 2026-11-02T14:37:12.518089Z info cache returned workload trust anchor from cache ttl=23h59m59.481911592s 2026-11-02T14:37:12.530329Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T14:37:12.743397Z info ads ADS: new connection for node:1 2026-11-02T14:37:12.743561Z info cache returned workload certificate from cache ttl=23h59m59.256441133s 2026-11-02T14:37:12.744010Z info ads ADS: new connection for node:2 2026-11-02T14:37:12.744097Z info cache returned workload trust anchor from cache ttl=23h59m59.255904025s 2026-11-02T14:37:12.863506Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:37:18.689911Z error failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused 2026-11-02T14:37:21.831139Z info Readiness succeeded in 9.392524483s 2026-11-02T14:37:21.831667Z info Envoy proxy is ready 2026-11-02T14:37:31.956023Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:38:51.746269Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:43:38.563978Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T14:43:38.564014Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T14:43:38.564028Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T14:43:38.564033Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-02T14:43:38.449Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 30 314 114 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "d8ed6707-07d0-4843-9253-fe7f53fe16d1" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.132.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.236:45868 10.133.0.236:443 10.132.0.13:35066 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T14:43:41.843459Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:44:08.068280Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:46:19.980804Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T14:46:30.508Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 73 294 85 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "18b8753f-2eaf-4202-944a-66e049e8ad10" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.132.0.41:8000" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.236:44334 10.133.0.236:443 10.132.0.13:36874 - v1-completions-path 2026-11-02T14:46:43.366240Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:46:45.705589Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:46:58.193919Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:47:02.397602Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:47:19.705682Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:47:40.143676Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:47:50.692022Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:48:01.049408Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:48:11.501481Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:48:21.903879Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:48:32.305857Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:48:42.690799Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:48:53.155767Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:49:03.566406Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:49:13.995317Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:49:24.197807Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:50:07.671206Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:50:23.551045Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:50:25.956524Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:50:29.394840Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-02T14:50:29.320Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 91 3616 74 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "56b3e915-25f3-469b-ab2a-627735eb58ab" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.134.0.41:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.236:49418 10.133.0.236:443 10.132.0.13:45818 - v1-catch-all-model-routing 2026-11-02T14:50:32.644078Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:50:39.500679Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:50:49.901557Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:51:04.953961Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:51:15.938704Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:51:17.984996Z info ads ADS: new connection for node:3 2026-11-02T14:51:17.985133Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T14:51:17.985153Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T14:51:20.025993Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:52:00.598837Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T14:52:18.802Z] "POST /v1/chat/completions HTTP/1.1" 403 - - "-" 131 0 36 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "805afb86-6fc6-4bdb-a77d-337d1f09ce6b" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "-" outbound|54321||e2e-prio-pool-170c8925-inference-pool-ip-0b8de156.llm.svc.cluster.local - 10.133.0.236:443 10.132.0.13:49804 - v1-model-routing 2026-11-02T14:52:20.103925Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:52:20.174976Z info ads ADS: "" 3 terminated 2026-11-02T14:52:22.659917Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:52:25.713933Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:52:31.437728Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:52:43.135467Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:53:00.113021Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:53:25.831971Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:53:40.222008Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:53:44.591839Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:53:56.440114Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:54:01.049976Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:54:16.775635Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:55:47.340050Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T14:55:47.340084Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T14:55:47.340098Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T14:55:47.340103Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T14:55:47.258Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 69 311 81 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "dddc1f7a-2249-4c44-ae51-f6b95e391367" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.132.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.236:43954 10.133.0.236:443 10.132.0.13:48406 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T14:58:57.417438Z info ads ADS: new connection for node:4 2026-11-02T14:58:57.423507Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-02T14:58:57.423559Z info cache read certificate from file resource=file-root:system 2026-11-02T14:58:58.595883Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:59:00.224109Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T14:59:11.384216Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-02T14:59:11.384366Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T14:59:11.384388Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T14:59:11.384397Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T14:59:11.384401Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T14:59:11.384551Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T14:59:11.268Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1720 116 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "7d2eeb21-971a-464b-84f0-e77c217074dd" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.236:59188 10.133.0.236:443 10.132.0.13:50558 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T14:59:39.526599Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T14:59:39.526632Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T14:59:39.526642Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T14:59:39.526646Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-02T14:59:39.526812Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-11-02T14:59:39.418Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1720 108 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "e4754262-0520-4d4c-bf6e-93b2f97aa40c" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.133.0.236:52088 10.133.0.236:443 10.132.0.13:60518 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T15:00:46.534673Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:00:46.712582Z info ads ADS: "" 4 terminated with stream closed 2026-11-02T15:00:48.124489Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:00:54.986308Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:01:10.723666Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:01:26.945490Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:01:31.829839Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:01:31.829876Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:01:31.829891Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:01:31.829895Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T15:01:31.752Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 323 77 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "42a36071-1608-49cf-89f7-c30e8e9b0523" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.132.0.32:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.236:57152 10.133.0.236:443 10.132.0.13:33670 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T15:01:50.425441Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:02:00.726078Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:02:05.513718Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T15:02:09.381Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 71 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "118b9e6c-d498-467c-ad3b-70fb3191eb51" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.134.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.236:40328 10.133.0.236:443 10.132.0.13:38998 - v1-completions-path 2026-11-02T15:02:22.602919Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:02:24.974065Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:02:32.624819Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T15:02:42.117Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 281 78 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "d2e25321-4e4e-4c60-a8d3-e2bfa109e7dc" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.134.0.43:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.236:60022 10.133.0.236:443 10.132.0.13:47074 - v1-completions-path 2026-11-02T15:02:44.696658Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:02:50.295366Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:02:58.303638Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:03:03.949551Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:03:21.962711Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:03:25.191091Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T15:03:28.334Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 295 66 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "4245a6c1-2da5-4f3e-8518-60e77a6c6df8" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.134.0.43:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.236:47378 10.133.0.236:443 10.132.0.13:48238 - v1-completions-path [2026-11-02T15:03:55.829Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 0 36 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "2ecd951a-3e5e-4dc5-b380-552203a2e7e7" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.236:443 10.132.0.13:37594 - v1-completions-path 2026-11-02T15:04:01.393852Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:04:19.604854Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:04:29.858349Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:07:17.798228Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:07:32.839555Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:07:45.297833Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:07:48.531563Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-02T15:07:48.684316Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T15:07:52.585Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 60 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "15952cba-1121-4b55-a429-527bf768b3fe" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.134.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.236:40328 10.133.0.236:443 10.132.0.13:57850 - v1-completions-path 2026-11-02T15:07:53.001976Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 2026-11-02T15:07:53.002054Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T15:07:53.002076Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T15:07:53.002093Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T15:07:53.002100Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-02T15:07:52.925Z] "DELETE /v1/api-keys/814154ed-079a-4591-b174-e2cbd69b5b4e HTTP/1.1" 200 - via_upstream - "-" 0 369 77 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "6d5251f6-30f0-45e4-b601-4e77a8bf716c" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.134.0.200:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.236:60252 10.133.0.236:443 10.132.0.13:57876 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T15:08:04.918218Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:08:15.281357Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:08:30.132772Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:08:35.311595Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T15:08:35.311631Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T15:08:35.311645Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-02T15:08:35.311650Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-02T15:08:35.065Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 59 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "66957c23-47c4-47da-878a-972a18b69a35" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.236:443 10.132.0.13:54082 - v1-completions-path [2026-11-02T15:08:35.235Z] "DELETE /v1/api-keys/77cf8adc-2e85-49e7-91f0-8aa978378b7e HTTP/1.1" 200 - via_upstream - "-" 0 365 76 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "36d07f70-1a22-4ea8-bc2c-9d80f7686e01" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.132.0.174:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.236:46386 10.133.0.236:443 10.132.0.13:54084 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T15:08:47.490131Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:08:52.135109Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:08:55.539911Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:09:01.599972Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:09:01.600015Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T15:09:01.600031Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T15:09:01.600035Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-02T15:09:01.521Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 68 311 78 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "5cd09cd0-e8f5-4d35-bd7d-ddf30b6f215a" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.132.0.174:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.133.0.236:46594 10.133.0.236:443 10.132.0.13:51750 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T15:09:01.625Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 60 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "c6e86b95-a03c-4e43-aec6-03b7474ce34a" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.134.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.236:56764 10.133.0.236:443 10.132.0.13:51762 - v1-completions-path [2026-11-02T15:09:02.149Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 58 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "ab8f10eb-70b7-4377-a5c9-a9fc2465c665" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.134.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.236:40328 10.133.0.236:443 10.132.0.13:51764 - v1-completions-path [2026-11-02T15:09:03.341Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "7edad548-6c1d-4d9d-8d6b-886230118140" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.134.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.236:56764 10.133.0.236:443 10.132.0.13:51774 - v1-completions-path [2026-11-02T15:09:04.437Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 59 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "35c6905e-fa77-411c-9a34-492e4b1d839e" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.134.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.236:56764 10.133.0.236:443 10.132.0.13:49356 - v1-completions-path [2026-11-02T15:09:04.974Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 58 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "1cb6a81e-d2d6-4688-b624-aa9d41e1e4ab" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.134.0.42:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.133.0.236:56764 10.133.0.236:443 10.132.0.13:49362 - v1-completions-path 2026-11-02T15:09:18.244649Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:09:28.630995Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:09:33.260525Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:09:45.660302Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:09:50.168417Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:10:08.542906Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:10:18.993174Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T15:10:33.563707Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T15:10:33.563732Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T15:10:33.563735Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T15:10:33.563739Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T15:10:33.513Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 49 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "a6a6d88b-bc10-4e49-a083-bc2520bd1de5" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.133.0.236:443 10.132.0.13:40320 - v1-completions-path [2026-11-02T15:10:35.694Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 70 - "34.228.250.194,10.132.0.13" "python-requests/2.33.0" "1a888c07-4028-4691-9ee2-bd44182b26c4" "maas.apps.ca30a64395dcfc3deb10.prod.konflux-ocp-ci.dev" "10.134.0.43:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.133.0.236:60022 10.133.0.236:443 10.132.0.13:54732 - v1-completions-path