2026-11-03T20:11:33.120202Z info FLAG: --concurrency="0" 2026-11-03T20:11:33.120220Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T20:11:33.120224Z info FLAG: --help="false" 2026-11-03T20:11:33.120227Z info FLAG: --log_as_json="false" 2026-11-03T20:11:33.120229Z info FLAG: --log_caller="" 2026-11-03T20:11:33.120231Z info FLAG: --log_output_level="default:info" 2026-11-03T20:11:33.120233Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T20:11:33.120239Z info FLAG: --log_target="[stdout]" 2026-11-03T20:11:33.120241Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T20:11:33.120243Z info FLAG: --outlierLogPath="" 2026-11-03T20:11:33.120245Z info FLAG: --profiling="true" 2026-11-03T20:11:33.120247Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T20:11:33.120249Z info FLAG: --proxyLogLevel="warning" 2026-11-03T20:11:33.120251Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T20:11:33.120253Z info FLAG: --stsPort="0" 2026-11-03T20:11:33.120255Z info FLAG: --templateFile="" 2026-11-03T20:11:33.120257Z info FLAG: --tokenManagerPlugin="" 2026-11-03T20:11:33.120259Z info FLAG: --vklog="0" 2026-11-03T20:11:33.120262Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T20:11:33.120404Z info Proxy role ips=[10.134.0.38] type=router id=maas-default-gateway-openshift-default-59f957fdd7-cd5ll.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T20:11:33.120450Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T20:11:33.121474Z info cpu limit detected as 2, setting concurrency 2026-11-03T20:11:33.121711Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T20:11:33.121722Z info JWT policy is third-party-jwt 2026-11-03T20:11:33.121725Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T20:11:33.122855Z info Starting default Istio SDS Server 2026-11-03T20:11:33.122880Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T20:11:33.122901Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T20:11:33.123231Z info Opening status port 15020 2026-11-03T20:11:33.124496Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T20:11:33.124779Z info sds Starting SDS grpc server 2026-11-03T20:11:33.124811Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T20:11:33.125785Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T20:11:33.126535Z info Starting proxy agent 2026-11-03T20:11:33.126566Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T20:11:33.200517Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T20:11:33.251927Z info cache generated new workload certificate resourceName=default latency=127.04348ms ttl=23h59m59.74807916s 2026-11-03T20:11:33.252074Z info cache Root cert has changed, start rotating root cert 2026-11-03T20:11:33.252157Z info cache returned workload trust anchor from cache ttl=23h59m59.747844538s 2026-11-03T20:11:33.317451Z info ads ADS: new connection for node:1 2026-11-03T20:11:33.317517Z info cache returned workload certificate from cache ttl=23h59m59.682484713s 2026-11-03T20:11:33.317671Z info ads ADS: new connection for node:2 2026-11-03T20:11:33.317751Z info cache returned workload trust anchor from cache ttl=23h59m59.682250539s 2026-11-03T20:11:33.418718Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:11:41.879816Z info Readiness succeeded in 8.762084216s 2026-11-03T20:11:41.880159Z info Envoy proxy is ready 2026-11-03T20:14:17.060118Z info ads ADS: new connection for node:3 2026-11-03T20:14:17.065415Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T20:14:17.065460Z info cache read certificate from file resource=file-root:system 2026-11-03T20:14:17.731085Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:14:29.435276Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:14:32.972567Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:14:32.972643Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:14:32.972657Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-03T20:14:32.972662Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:14:32.972838Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-11-03T20:14:32.893Z] "POST /llm/e2e-rt-sw-9dcd2a/anything/e2e-rt-sw-9dcd2a-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 79 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "3af1ec3a-613e-40f2-adc5-6da5d2749f30" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-9dcd2a.llm.svc.cluster.local - 10.134.0.38:443 10.132.0.13:58238 - llm.external-model-e2e-rt-sw-9dcd2a-914404c28491.2 2026-11-03T20:14:37.057949Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:14:58.242207Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:14:58.242235Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:14:58.242244Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-03T20:14:58.242248Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:14:58.242420Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-11-03T20:14:58.103Z] "POST /anything/e2e-rt-sw-9dcd2a-client HTTP/1.1" 200 - via_upstream - "-" 124 1584 139 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "d9a3bfc6-1ace-4b9c-b6fa-87343b908876" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.38:60520 10.134.0.38:443 10.132.0.13:37620 - llm.external-model-e2e-rt-sw-9dcd2a-914404c28491.0 2026-11-03T20:15:05.997354Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:15:15.791993Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:15:15.792037Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:15:15.792052Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-03T20:15:15.792057Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:15:15.792289Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-11-03T20:15:15.605Z] "POST /llm/e2e-rt-sw-9dcd2a/anything/e2e-rt-sw-9dcd2a-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 186 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "c8bafee2-b190-45df-acfb-1de4626ad046" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-9dcd2a.llm.svc.cluster.local - 10.134.0.38:443 10.132.0.13:56018 - llm.external-model-e2e-rt-sw-9dcd2a-914404c28491.2 2026-11-03T20:15:36.785802Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:15:53.220171Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:16:22.519803Z info ads ADS: "" 3 terminated 2026-11-03T20:16:22.833695Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:16:26.175303Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:16:49.115065Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:17:10.657956Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:17:32.983250Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:18:35.329376Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:18:35.329413Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:18:35.329426Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T20:18:35.329430Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T20:18:35.197Z] "DELETE /v1/api-keys/0e3a43ce-5672-4228-8b9d-15efe0c3b975 HTTP/1.1" 200 - via_upstream - "-" 0 370 132 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "7282bf3c-f947-412b-a72e-e995bcb46301" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.38:55416 10.134.0.38:443 10.132.0.13:48102 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T20:18:36.560050Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:18:36.560151Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=26 2026-11-03T20:18:36.560204Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:18:36.560257Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T20:18:36.560291Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 [2026-11-03T20:18:36.473Z] "DELETE /v1/api-keys/98491fc3-c9cf-4b66-af40-66fc017c9e3e HTTP/1.1" 200 - via_upstream - "-" 0 321 87 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "72df3604-584e-4fe4-82df-a6afd476a1ee" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.38:55426 10.134.0.38:443 10.132.0.13:48108 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T20:18:40.849296Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:18:40.849327Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:18:40.849339Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:18:40.849344Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T20:18:40.787Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 62 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "105b86b1-39e8-439d-a1d8-4a12c3aabb57" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.38:55416 10.134.0.38:443 10.132.0.13:48122 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T20:19:14.650772Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T20:20:50.290Z] "POST /maas-api/internal/v1/subscriptions/select HTTP/1.1" 404 NR route_not_found - "-" 56 0 13 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "f86cb9bc-1de8-4efc-89d7-6e7482b7a168" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "-" - - 10.134.0.38:443 10.132.0.13:38616 - - 2026-11-03T20:24:06.969799Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:24:06.969839Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:24:06.969857Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T20:24:06.969864Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-03T20:24:06.901Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 86 328 68 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "b2abe9bb-a788-49b1-9968-e4ca490ac97d" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.38:45762 10.134.0.38:443 10.132.0.13:44750 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T20:24:32.568532Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:24:32.568579Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:24:32.568613Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T20:24:32.568621Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T20:24:32.497Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 70 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "294301c1-1aff-449e-946f-20a79c1ee92c" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.38:45762 10.134.0.38:443 10.132.0.13:51536 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T20:33:14.605785Z info ads ADS: new connection for node:4 2026-11-03T20:33:14.631562Z info cache read certificate from file resource=file-root:system 2026-11-03T20:40:06.481556Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:40:43.432002Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:41:07.186567Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:41:09.236364Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-03T20:41:09.641491Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:41:10.403793Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:41:16.900847Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:41:22.908876Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:41:54.886527Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:41:55.812943Z info ads ADS: new connection for node:5 2026-11-03T20:41:55.813232Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T20:41:55.813313Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T20:41:59.903313Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:42:37.061582Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:43:13.806475Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:43:35.130044Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:43:47.661722Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:43:48.437123Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:43:55.442721Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:46:45.988206Z info ads ADS: "" 5 terminated with stream closed 2026-11-03T20:51:11.086658Z info ads ADS: "" 4 terminated with stream closed 2026-11-03T20:52:08.677693Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:53:47.943045Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:54:58.644591Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T20:56:35.978503Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:00:38.101402Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-11-03T21:02:55.253650Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-11-03T21:06:48.826561Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:07:12.058657Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:07:46.284228Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:07:49.907720Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-03T21:07:49.750Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 108118 108394 160 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "5f4507ff-7a1b-4f56-a177-1b4dea4db1ef" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.133.0.37:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.134.0.38:46958 10.134.0.38:443 10.132.0.13:39796 - v1-chat-completions-path 2026-11-03T21:08:02.300168Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:08:04.140069Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:08:07.489978Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=26 [2026-11-03T21:08:07.402Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 87 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "7bc9e28b-596c-4ec8-94ef-144c25fdddf8" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.38:53378 10.134.0.38:443 10.132.0.13:41368 - v1-catch-all-path 2026-11-03T21:08:10.462692Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:08:14.307250Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=26 [2026-11-03T21:08:14.246Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 60 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "01642656-b01f-4d96-b580-5b8038c4c153" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.38:53378 10.134.0.38:443 10.132.0.13:39392 - v1-catch-all-path 2026-11-03T21:08:26.140982Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:08:27.955514Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:08:42.932934Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:08:53.867672Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:08:56.522652Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:08:56.559258Z info ads ADS: new connection for node:6 2026-11-03T21:08:56.559519Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T21:08:56.559555Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T21:08:56.865986Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:09:00.072141Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:09:00.072181Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:09:00.072198Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:09:00.072206Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T21:08:59.864Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 90 332 207 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "2da895d6-2a75-4b1c-b1e2-d8184528ac6a" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.38:60332 10.134.0.38:443 10.132.0.13:33514 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T21:09:11.674199Z info ads ADS: "" 6 terminated 2026-11-03T21:09:11.722678Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:09:12.191138Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:09:15.495882Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:09:15.986571Z info ads ADS: new connection for node:7 2026-11-03T21:09:15.986849Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-03T21:09:17.816298Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:09:56.612239Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T21:10:02.254Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 331 90 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "93f31e72-3fb0-442a-8a50-38d93202d226" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.133.0.188:8000" outbound|54321||e2e-prio-pool-0704dddb-inference-pool-ip-461f2a7f.llm.svc.cluster.local 10.134.0.38:41554 10.134.0.38:443 10.132.0.13:54594 - v1-model-routing 2026-11-03T21:10:16.047739Z info ads ADS: "" 7 terminated 2026-11-03T21:10:16.171527Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:10:18.077733Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:10:20.742837Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:10:25.465772Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:12:39.240742Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:12:55.081229Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:13:10.728586Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:13:22.476867Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:13:26.516088Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:13:37.844986Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:13:40.452058Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:13:55.930148Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T21:14:11.340Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 74 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "84c921d2-7a2e-4d8d-b69a-04b3f19da835" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.38:39854 10.134.0.38:443 10.132.0.13:47352 - v1-completions-path 2026-11-03T21:14:40.525336Z info ads ADS: new connection for node:8 2026-11-03T21:14:40.530710Z info cache read certificate from file resource=file-root:system 2026-11-03T21:14:41.091988Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:14:42.935039Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:14:52.730826Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"983","StreamId":"2649062880737315163"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-11-03T21:14:52.669Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 61 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "90da98e1-5a5e-4816-9664-0e8bcdc0033c" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.134.0.38:443 10.132.0.13:47370 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-03T21:15:01.534688Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T21:15:01.534720Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T21:15:01.534731Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-03T21:15:01.534736Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T21:15:01.534911Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-11-03T21:15:01.405Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1535 129 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "236fc471-a6c0-4d41-ace8-cbe13281c149" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.38:43024 10.134.0.38:443 10.132.0.13:38066 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T21:15:03.681348Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T21:15:03.681378Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T21:15:03.681388Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T21:15:03.681392Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-03T21:15:03.681534Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=26 [2026-11-03T21:15:03.588Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1536 93 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "cc23a7a3-3718-4430-8c8a-0c4d49ee3b0e" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.38:43024 10.134.0.38:443 10.132.0.13:38078 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T21:15:49.598782Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:15:49.598819Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:15:49.598835Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:15:49.598840Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T21:15:49.599046Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T21:15:49.483Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1529 115 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "d08f76b5-7dcc-40d4-9847-6b334cf0f44f" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.38:35576 10.134.0.38:443 10.132.0.13:34240 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T21:16:00.397787Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:16:00.397842Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:16:00.397858Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:16:00.397863Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T21:16:00.398120Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-03T21:16:00.300Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1535 97 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "ccbac918-c6cc-44a8-b4bc-0e4b26c30c41" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.38:35576 10.134.0.38:443 10.132.0.13:53890 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-03T21:16:17.903876Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:16:18.057142Z info ads ADS: "" 8 terminated 2026-11-03T21:16:18.764441Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:16:22.034555Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:16:25.365003Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T21:16:24.850Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 59 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "c509edb3-20dd-46e4-9af9-3f536a3f87f6" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.38:443 10.132.0.13:35394 - v1-completions-path 2026-11-03T21:16:36.778768Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:16:38.713134Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:16:54.019023Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:16:55.965814Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:16:59.255287Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:17:06.829025Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:17:12.501766Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T21:17:12.501812Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T21:17:12.501831Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-03T21:17:12.501838Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-11-03T21:17:12.444Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 77 319 57 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "7df23e57-0fb7-4b32-9d3b-b814603f7f70" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.38:48340 10.134.0.38:443 10.132.0.13:46808 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T21:17:14.177Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 72 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "eda8d67a-3470-4ebc-a808-4e20f535bae0" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.38:43952 10.134.0.38:443 10.132.0.13:41156 - v1-completions-path [2026-11-03T21:17:14.447Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 57 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "9286c1f9-99e6-4a73-a3c4-3fafcaeaab65" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.38:43952 10.134.0.38:443 10.132.0.13:41164 - v1-completions-path [2026-11-03T21:17:15.144Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 58 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "9a8686bb-c191-467e-bb41-227d57aee6e3" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.38:43952 10.134.0.38:443 10.132.0.13:41184 - v1-completions-path 2026-11-03T21:17:30.986436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:17:41.287154Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:17:43.198168Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T21:17:48.844Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 60 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "64393534-03e5-468b-b747-8407a07f8836" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.38:43952 10.134.0.38:443 10.132.0.13:47208 - v1-completions-path 2026-11-03T21:18:02.099773Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:18:04.063672Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:18:10.123210Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:18:21.032119Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:18:34.689085Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:18:36.520419Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:18:42.504484Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:19:00.115855Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:19:03.201795Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:19:18.752409Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:19:21.208926Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:19:28.547651Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:19:34.681730Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:19:34.681768Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:19:34.681785Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:19:34.681792Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T21:19:34.622Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 309 59 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "f6fbcde0-7673-4a81-9908-17b28661aaa9" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.29:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.38:46302 10.134.0.38:443 10.132.0.13:38334 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T21:19:46.851564Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:19:57.101537Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:20:13.463766Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:20:16.640525Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:24:04.096344Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:24:19.211938Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:24:21.048114Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:24:28.306081Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:24:46.468974Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:24:50.251836Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:24:53.791296Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T21:24:53.791395Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 2026-11-03T21:24:53.791434Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=26 2026-11-03T21:24:53.791442Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=26 [2026-11-03T21:24:53.616Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 314 175 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "027d02b2-4179-4451-960c-24218834c786" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.1.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.38:33756 10.134.0.38:443 10.132.0.13:49660 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T21:24:54.643Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 290 64 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "490e3664-c338-44ca-b685-279f985d5bd6" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.38:39630 10.134.0.38:443 10.132.0.13:39736 - v1-completions-path 2026-11-03T21:25:05.982048Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:25:16.218002Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:25:18.727632Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:25:39.166347Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:25:41.594251Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:25:45.596292Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:25:54.984446Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:25:54.984482Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:25:54.984497Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T21:25:54.984502Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T21:25:54.924Z] "DELETE /v1/api-keys/36e1901e-55cd-4e60-bc0e-9ce67af9c800 HTTP/1.1" 200 - via_upstream - "-" 0 365 60 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "d4774ea4-baa2-489f-a7b9-5834028f0136" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.1.30:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.38:41984 10.134.0.38:443 10.132.0.13:60912 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T21:26:06.601536Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:26:16.821377Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:26:19.228056Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:26:23.200255Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:26:28.615646Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T21:26:33.728Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 68 - "44.212.242.249,10.132.0.13" "python-requests/2.34.2" "fdc7ae40-03ac-4c3f-8c7e-21fc5aaab601" "maas.apps.96089cf8aa8b05683edc.prod.konflux-ocp-ci.dev" "10.134.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.38:43952 10.134.0.38:443 10.132.0.13:49258 - v1-completions-path 2026-11-03T21:26:47.208851Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:26:57.381346Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:26:59.265690Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:27:09.224923Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:27:13.119285Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:27:17.880177Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:27:35.300095Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:27:55.096489Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:28:05.436154Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:28:15.534404Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:28:25.742745Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:28:35.952926Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:28:46.240822Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:28:56.324664Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:29:06.536837Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:29:16.754759Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T21:29:26.959774Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8