2026-10-30T16:31:44.690710Z info FLAG: --concurrency="0" 2026-10-30T16:31:44.690801Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-10-30T16:31:44.690807Z info FLAG: --help="false" 2026-10-30T16:31:44.690810Z info FLAG: --log_as_json="false" 2026-10-30T16:31:44.690813Z info FLAG: --log_caller="" 2026-10-30T16:31:44.690816Z info FLAG: --log_output_level="default:info" 2026-10-30T16:31:44.690819Z info FLAG: --log_stacktrace_level="default:none" 2026-10-30T16:31:44.690826Z info FLAG: --log_target="[stdout]" 2026-10-30T16:31:44.690830Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-10-30T16:31:44.690833Z info FLAG: --outlierLogPath="" 2026-10-30T16:31:44.690835Z info FLAG: --profiling="true" 2026-10-30T16:31:44.690838Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-10-30T16:31:44.690841Z info FLAG: --proxyLogLevel="warning" 2026-10-30T16:31:44.690844Z info FLAG: --serviceCluster="istio-proxy" 2026-10-30T16:31:44.690848Z info FLAG: --stsPort="0" 2026-10-30T16:31:44.690850Z info FLAG: --templateFile="" 2026-10-30T16:31:44.690853Z info FLAG: --tokenManagerPlugin="" 2026-10-30T16:31:44.690857Z info FLAG: --vklog="0" 2026-10-30T16:31:44.690860Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-10-30T16:31:44.691042Z info Proxy role ips=[10.132.0.46] type=router id=maas-default-gateway-openshift-default-7854949c9-smq4m.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-10-30T16:31:44.691107Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-10-30T16:31:44.692295Z info cpu limit detected as 2, setting concurrency 2026-10-30T16:31:44.692492Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-10-30T16:31:44.692503Z info JWT policy is third-party-jwt 2026-10-30T16:31:44.692506Z info using credential fetcher of JWT type in cluster.local trust domain 2026-10-30T16:31:44.694312Z info Starting default Istio SDS Server 2026-10-30T16:31:44.694335Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-10-30T16:31:44.694367Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-10-30T16:31:44.695003Z info Opening status port 15020 2026-10-30T16:31:44.696507Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-10-30T16:31:44.698219Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-10-30T16:31:44.699305Z info Starting proxy agent 2026-10-30T16:31:44.699345Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-10-30T16:31:44.706228Z info sds Starting SDS grpc server 2026-10-30T16:31:44.706248Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-10-30T16:31:44.773421Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-10-30T16:31:44.797470Z info cache generated new workload certificate resourceName=default latency=100.776434ms ttl=23h59m59.202535278s 2026-10-30T16:31:44.797530Z info cache Root cert has changed, start rotating root cert 2026-10-30T16:31:44.797563Z info cache returned workload trust anchor from cache ttl=23h59m59.202437354s 2026-10-30T16:31:44.927910Z info ads ADS: new connection for node:1 2026-10-30T16:31:44.927996Z info cache returned workload certificate from cache ttl=23h59m59.072006464s 2026-10-30T16:31:44.928093Z info ads ADS: new connection for node:2 2026-10-30T16:31:44.928171Z info cache returned workload trust anchor from cache ttl=23h59m59.071829989s 2026-10-30T16:31:45.017752Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T16:31:52.959956Z info Readiness succeeded in 8.272085923s 2026-10-30T16:31:52.960439Z info Envoy proxy is ready [2026-10-30T16:31:56.163Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 406 82 - "52.71.36.72,10.132.0.10" "curl/7.76.1" "557b06d1-d07c-4ffe-9cd2-0f3fdb98582d" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.46:50390 10.132.0.46:443 10.132.0.10:54402 - v1-model-routing [2026-10-30T16:31:56.658Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 58 - "52.71.36.72,10.132.0.10" "curl/7.76.1" "2a06bb33-dde6-423d-8b4a-1b12c142657d" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.46:443 10.132.0.10:53438 - v1-model-routing [2026-10-30T16:31:56.824Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 50 - "52.71.36.72,10.132.0.10" "curl/7.76.1" "d0391dbe-8cd9-4127-b7ee-d10a52a01e3b" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.46:443 10.132.0.10:53444 - v1-model-routing 2026-10-30T16:31:57.226840Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T16:31:57.226862Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T16:31:57.226865Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-10-30T16:31:57.226868Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 2026-10-30T16:31:57.339243Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:31:57.339277Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:31:57.339291Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T16:31:57.339296Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T16:31:56.899Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 57 - "52.71.36.72,10.132.0.10" "curl/7.76.1" "c551d1d9-27bd-49e2-95db-722463372850" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.46:443 10.132.0.10:53446 - v1-model-routing [2026-10-30T16:31:57.059Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 50 - "52.71.36.72,10.132.0.10" "curl/7.76.1" "b279fe1f-61e8-4802-9bb6-bca0344a0d33" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.46:443 10.132.0.10:53450 - v1-model-routing [2026-10-30T16:31:57.132Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 53 - "52.71.36.72,10.132.0.10" "curl/7.76.1" "1fb1918e-3189-4302-a85b-c64445030dfb" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.46:443 10.132.0.10:53464 - v1-model-routing [2026-10-30T16:31:57.210Z] "POST /v1/chat/completions HTTP/1.1" 401 - - "-" 122 88 16 - "52.71.36.72,10.132.0.10" "curl/7.76.1" "a9c756bd-9010-4ce3-b9de-7a3fe37e674a" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.46:443 10.132.0.10:53466 - v1-model-routing [2026-10-30T16:31:57.265Z] "DELETE /v1/api-keys/06ee18a8-d075-427b-89fb-c390e46acd4f HTTP/1.1" 200 - via_upstream - "-" 0 365 73 - "52.71.36.72,10.132.0.10" "curl/7.76.1" "e4717c89-ea39-4099-b256-1020bdfdb998" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.132.0.37:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.46:37662 10.132.0.46:443 10.132.0.10:53478 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T16:35:15.305602Z info ads ADS: new connection for node:3 2026-10-30T16:35:15.311349Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-10-30T16:35:15.311414Z info cache read certificate from file resource=file-root:system 2026-10-30T16:35:16.998875Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T16:35:21.830524Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T16:35:32.868960Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T16:35:35.684356Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:35.684400Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:35.684420Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T16:35:35.684428Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T16:35:35.459Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 74 316 224 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "584164c6-72a8-408e-959c-d99d3cf1289d" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.132.0.37:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.46:33464 10.132.0.46:443 10.132.0.10:38446 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T16:35:36.756857Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:36.756917Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:36.756933Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:36.756960Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T16:35:36.757259Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T16:35:36.478Z] "POST /anything/e2e-rt-sw-20ea01-client HTTP/1.1" 200 - via_upstream - "-" 124 1758 278 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "a0704260-e797-443b-bf8f-4ec9de1c3fd0" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.46:52264 10.132.0.46:443 10.132.0.10:50532 - llm.external-model-e2e-rt-sw-20ea01-9b805744198a.2 2026-10-30T16:35:37.565276Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:37.565306Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:37.565317Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T16:35:37.565321Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:37.565503Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-10-30T16:35:36.779Z] "POST /anything/e2e-rt-sw-20ea01-client HTTP/1.1" 200 - via_upstream - "-" 124 1750 786 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "a3fa4d5f-d564-9519-bca1-1e5e943eff29" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.46:52268 10.132.0.46:443 10.132.0.10:50544 - llm.external-model-e2e-rt-sw-20ea01-9b805744198a.2 2026-10-30T16:35:42.948146Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:42.948177Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:42.948187Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:42.948191Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T16:35:42.948339Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 2026-10-30T16:35:43.164070Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T16:35:42.781Z] "POST /anything/e2e-rt-sw-20ea01-client HTTP/1.1" 200 - via_upstream - "-" 124 1751 167 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "8850767d-edd4-4fc3-b059-5e473be29e6f" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.46:52274 10.132.0.46:443 10.132.0.10:50562 - llm.external-model-e2e-rt-sw-20ea01-9b805744198a.0 2026-10-30T16:35:53.414065Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-10-30T16:35:53.469207Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:53.469235Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:53.469246Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T16:35:53.469250Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:35:53.469387Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 2026-10-30T16:35:53.913706Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T16:35:53.405Z] "POST /llm/e2e-rt-sw-20ea01/anything/e2e-rt-sw-20ea01-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 63 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "82fbc068-511f-4b35-bfb3-a41e7e81cecf" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-20ea01.llm.svc.cluster.local - 10.132.0.46:443 10.132.0.10:51302 - llm.external-model-e2e-rt-sw-20ea01-9b805744198a.2 2026-10-30T16:36:01.820418Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:36:01.820452Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:36:01.820464Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:36:01.820468Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T16:36:01.820621Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 2026-10-30T16:36:02.426437Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-10-30T16:35:58.674Z] "POST /anything/e2e-rt-sw-20ea01-client HTTP/1.1" 200 - via_upstream - "-" 124 1756 3146 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "d335bbc9-ccce-42e7-ae94-c359f2519918" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.46:43444 10.132.0.46:443 10.132.0.10:39094 - llm.external-model-e2e-rt-sw-20ea01-9b805744198a.0 2026-10-30T16:36:15.347772Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T16:36:31.090856Z info ads ADS: "" 3 terminated 2026-10-30T16:36:31.825159Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T16:36:48.384897Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T16:38:38.281525Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T16:39:02.519211Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T16:40:35.483581Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T16:42:28.606853Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:42:28.606898Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:42:28.606917Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T16:42:28.606922Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T16:42:28.508Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 98 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "30cc70d3-05b8-40fc-8b67-976bc63d8899" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.132.0.37:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.46:48812 10.132.0.46:443 10.132.0.10:57318 - odh-ai-gateway-infra.maas-api-route.3 2026-10-30T16:42:29.342237Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:42:29.342275Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:42:29.342291Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T16:42:29.342296Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:42:30.014989Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:42:30.015052Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:42:30.015080Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:42:30.015088Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T16:42:29.157Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 184 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "b16f37d8-012c-4441-999d-054d961a27bc" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.132.0.37:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.46:48812 10.132.0.46:443 10.132.0.10:57328 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T16:42:29.870Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 144 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "12788c0d-9e76-97c7-bf13-40083f9c9f30" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.132.0.37:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.46:48812 10.132.0.46:443 10.132.0.10:57340 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T16:42:39.022Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 290 64 - "52.71.36.72,10.132.0.10" "-" "8c487ba4-e875-4edf-b72c-157c5b7aa984" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.46:50390 10.132.0.46:443 10.132.0.10:42594 - v1-completions-path 2026-10-30T16:42:56.731670Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T16:43:22.300743Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T16:43:23.558360Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T16:43:23.558398Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T16:43:23.558413Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T16:43:23.558418Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-10-30T16:43:23.437Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 120 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "a48c4915-c4e9-49e0-bd7e-82c402513dfc" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.132.0.37:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.46:60140 10.132.0.46:443 10.132.0.10:33630 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T16:43:23.875Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 76 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "0890c214-a693-46b9-b07f-9515ca270cef" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.46:51972 10.132.0.46:443 10.132.0.10:33634 - v1-completions-path 2026-10-30T16:43:24.380439Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:43:24.380473Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:43:24.380486Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T16:43:24.380491Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-10-30T16:43:24.373Z] "GET /health HTTP/1.1" 200 - via_upstream - "-" 0 91 7 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "dda1ce06-860e-4225-81a1-0a23aec70493" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.132.0.37:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.46:48812 10.132.0.46:443 10.132.0.10:33642 - odh-ai-gateway-infra.maas-api-route.4 2026-10-30T16:47:36.207027Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:47:36.207061Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:47:36.207074Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:47:36.207079Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T16:47:36.207234Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 2026-10-30T16:47:36.207290Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T16:47:36.207298Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T16:47:36.207301Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-10-30T16:47:36.207304Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-10-30T16:47:36.125Z] "POST /v1/chat/completions HTTP/1.1" 404 - via_upstream - "-" 120 126 82 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "77fe2e94-9a13-4075-a87c-d3d2c1fcc1c7" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.46:50390 10.132.0.46:443 10.132.0.10:41312 - v1-chat-completions-path 2026-10-30T16:50:32.570193Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:50:32.570231Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:50:32.570247Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-10-30T16:50:32.570252Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:50:32.665283Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:50:32.665315Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:50:32.665329Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-10-30T16:50:32.665334Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-10-30T16:50:32.495Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 74 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "7e48799f-49a7-4929-bb6f-ae9286824509" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.132.0.37:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.46:36012 10.132.0.46:443 10.132.0.10:47072 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T16:50:32.592Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 79 322 72 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "c281450d-8a37-4d86-929a-780e45d2d3ef" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.132.0.37:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.46:36012 10.132.0.46:443 10.132.0.10:47088 - odh-ai-gateway-infra.maas-api-route.3 [2026-10-30T16:50:32.685Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 136 376 68 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "660b2676-f4a7-49e7-b005-1fc344816a86" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "10.133.0.36:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.46:50390 10.132.0.46:443 10.132.0.10:47098 - v1-chat-completions-path [2026-10-30T16:59:43.919Z] "GET /v1/tenants HTTP/1.1" 404 NR route_not_found - "-" 0 0 11 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "93732756-01d9-429f-94ee-b9aba1ceab13" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "-" - - 10.132.0.46:443 10.132.0.10:33642 - - 2026-10-30T17:02:29.240193Z info ads ADS: new connection for node:4 2026-10-30T17:02:29.250116Z info cache read certificate from file resource=file-root:system 2026-10-30T17:04:33.418141Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-10-30T17:09:34.167129Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T17:10:01.001599Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T17:10:03.844388Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T17:10:03.844426Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T17:10:03.844441Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-10-30T17:10:03.844448Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-10-30T17:10:03.844701Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-10-30T17:10:03.687Z] "POST /anything/e2e-dft-2ad5de-1f59e3 HTTP/1.1" 200 - via_upstream - "-" 109 1745 156 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "ad66ecb2-8676-433a-8687-6e5c736fbe2d" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.46:47496 10.132.0.46:443 10.132.0.10:40860 - llm.external-model-e2e-dft-2ad5de-6844278b802b.0 [2026-10-30T17:10:03.927Z] "POST /llm/e2e-dft-2ad5de/anything/e2e-dft-2ad5de-1f59e3 HTTP/1.1" 401 - - "-" 109 88 89 - "52.71.36.72,10.132.0.10" "python-requests/2.34.2" "2889ecea-7872-43a9-a519-794782f787e4" "maas.apps.916cfd10db9db2c980a8.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-2ad5de.llm.svc.cluster.local - 10.132.0.46:443 10.132.0.10:40862 - llm.external-model-e2e-dft-2ad5de-6844278b802b.2 2026-10-30T17:10:24.412701Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T17:10:30.917569Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T17:10:36.555176Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T17:10:49.760384Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T17:10:57.635937Z info ads ADS: new connection for node:5 2026-10-30T17:10:57.636111Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T17:10:57.636168Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-10-30T17:11:29.426876Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-10-30T17:13:51.034416Z info ads ADS: "" 5 terminated 2026-10-30T17:16:00.123969Z info ads ADS: "" 4 terminated 2026-10-30T17:20:00.940213Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy