2026-11-02T22:07:34.894647Z info FLAG: --concurrency="0" 2026-11-02T22:07:34.894689Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T22:07:34.894695Z info FLAG: --help="false" 2026-11-02T22:07:34.894699Z info FLAG: --log_as_json="false" 2026-11-02T22:07:34.894702Z info FLAG: --log_caller="" 2026-11-02T22:07:34.894706Z info FLAG: --log_output_level="default:info" 2026-11-02T22:07:34.894709Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T22:07:34.894716Z info FLAG: --log_target="[stdout]" 2026-11-02T22:07:34.894720Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T22:07:34.894723Z info FLAG: --outlierLogPath="" 2026-11-02T22:07:34.894726Z info FLAG: --profiling="true" 2026-11-02T22:07:34.894730Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T22:07:34.894733Z info FLAG: --proxyLogLevel="warning" 2026-11-02T22:07:34.894736Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T22:07:34.894740Z info FLAG: --stsPort="0" 2026-11-02T22:07:34.894743Z info FLAG: --templateFile="" 2026-11-02T22:07:34.894747Z info FLAG: --tokenManagerPlugin="" 2026-11-02T22:07:34.894751Z info FLAG: --vklog="0" 2026-11-02T22:07:34.894755Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T22:07:34.894930Z info Proxy role ips=[10.134.0.35] type=router id=maas-default-gateway-openshift-default-696f47c6c-xlsg6.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T22:07:34.894990Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T22:07:34.898826Z info cpu limit detected as 2, setting concurrency 2026-11-02T22:07:34.899607Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T22:07:34.899623Z info JWT policy is third-party-jwt 2026-11-02T22:07:34.899628Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T22:07:34.900910Z info Starting default Istio SDS Server 2026-11-02T22:07:34.900930Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T22:07:34.900954Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T22:07:34.901059Z info Opening status port 15020 2026-11-02T22:07:34.901661Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T22:07:34.902463Z info sds Starting SDS grpc server 2026-11-02T22:07:34.902485Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T22:07:34.903960Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T22:07:34.905132Z info Starting proxy agent 2026-11-02T22:07:34.905210Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T22:07:34.976492Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T22:07:35.112088Z info ads ADS: new connection for node:1 2026-11-02T22:07:35.112622Z info ads ADS: new connection for node:2 2026-11-02T22:07:35.353133Z info cache generated new workload certificate resourceName=default latency=450.465694ms ttl=23h59m59.646871935s 2026-11-02T22:07:35.353181Z info cache Root cert has changed, start rotating root cert 2026-11-02T22:07:35.353229Z info cache returned workload trust anchor from cache ttl=23h59m59.646772187s 2026-11-02T22:07:35.353258Z info cache returned workload certificate from cache ttl=23h59m59.646746036s 2026-11-02T22:07:35.353365Z info cache returned workload trust anchor from cache ttl=23h59m59.646636131s 2026-11-02T22:07:35.353739Z info cache returned workload trust anchor from cache ttl=23h59m59.646263275s 2026-11-02T22:07:35.437045Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:07:43.333015Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:07:44.495931Z warn Envoy proxy is NOT ready: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-11-02T22:07:45.282791Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:07:45.313993Z info Readiness succeeded in 10.423380793s 2026-11-02T22:07:45.314345Z info Envoy proxy is ready 2026-11-02T22:08:11.643764Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:08:11.643799Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:08:11.643813Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T22:08:11.643818Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-02T22:08:11.558Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 55 321 85 - "44.212.242.249,10.133.0.11" "curl/7.76.1" "aa0aac97-b482-44ba-bb4c-e4e14950e09d" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.35:53820 10.134.0.35:443 10.133.0.11:39898 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T22:08:13.319Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 55 - "44.212.242.249,10.133.0.11" "curl/7.76.1" "839700f9-a23d-43ca-a95c-7c2dec3304a9" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.35:443 10.133.0.11:39900 - v1-model-routing 2026-11-02T22:10:19.331977Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:19.332020Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:19.332034Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:19.332039Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T22:10:19.629600Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:10:19.629636Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:10:19.629650Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T22:10:19.629654Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:10:19.863738Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:10:19.863770Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:10:19.863784Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T22:10:19.863788Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-02T22:10:19.226Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 84 326 105 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "679e6f76-7932-415b-a927-65ccaa7e1077" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.35:43028 10.134.0.35:443 10.133.0.11:39036 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T22:10:19.517Z] "DELETE /v1/api-keys/9f6c6358-0ef6-4a3f-a78f-f53cad814560 HTTP/1.1" 200 - via_upstream - "-" 0 370 112 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "eb9d8d66-4dab-4aca-ac6b-f78c783dba68" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.35:43034 10.134.0.35:443 10.133.0.11:39040 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T22:10:19.789Z] "DELETE /v1/api-keys/70cf618b-b31a-4ea6-b411-92a4feadf2c1 HTTP/1.1" 200 - via_upstream - "-" 0 321 73 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "642c77ad-3af0-4d3b-a95a-55ddd70eee61" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.35:43034 10.134.0.35:443 10.133.0.11:39042 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T22:10:21.351942Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:21.352014Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:21.352045Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:10:21.352052Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T22:10:21.236Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 115 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "752c74bb-c0b2-4617-86e9-d72ad81372d3" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.35:43028 10.134.0.35:443 10.133.0.11:51872 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T22:10:28.670587Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:10:28.670633Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:10:28.670649Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T22:10:28.670654Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-02T22:10:28.585Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 85 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "365ee7f8-6581-4f52-bb48-73ce9d95274d" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.35:43034 10.134.0.35:443 10.133.0.11:51888 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T22:10:30.540921Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 [2026-11-02T22:10:30.457Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 287 84 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "1fad34ca-7732-4639-a2dd-d0fd45f33694" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.134.0.32:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.35:53308 10.134.0.35:443 10.133.0.11:51904 - v1-completions-path 2026-11-02T22:10:37.108300Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:10:51.860237Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:14:16.640727Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:14:28.039585Z warn xdsproxy sendDownstreamDelta took 10.987592241s 2026-11-02T22:15:23.417857Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:16:34.531102Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:17:16.160962Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:17:54.363948Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:18:37.693477Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:19:52.142651Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:20:03.704261Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-02T22:20:22.907607Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:26:31.502280Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-11-02T22:27:20.228759Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:27:45.944954Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:28:29.788527Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T22:28:29.788554Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T22:28:29.788558Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T22:28:29.788562Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-02T22:28:29.755Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 33 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "f2b8c20f-142d-40d0-886a-98c85db17527" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.35:443 10.133.0.11:34248 - v1-completions-path 2026-11-02T22:28:32.024436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:28:47.438131Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:28:49.033930Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T22:28:54.538Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 56 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "b4b0acd7-ee7c-4092-b209-ce1e8c9ae2dc" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.35:443 10.133.0.11:38734 - v1-completions-path 2026-11-02T22:29:06.336937Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:29:08.012115Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:29:10.627406Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:29:19.099300Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:29:21.863074Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:29:21.863109Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T22:29:21.863127Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T22:29:21.863131Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-02T22:29:21.753Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 81 324 109 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "fb877a86-5a93-47fb-82c3-86ca0051ded4" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.35:44754 10.134.0.35:443 10.133.0.11:53484 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T22:29:23.521512Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 [2026-11-02T22:29:23.218Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 71 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "adedf580-9804-4340-92a3-bb1b029c1355" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.35:48162 10.134.0.35:443 10.133.0.11:53490 - v1-completions-path [2026-11-02T22:29:23.466Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 59 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "48617a70-f5b1-4327-9291-1a1e49439a5e" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.35:48162 10.134.0.35:443 10.133.0.11:53498 - v1-completions-path [2026-11-02T22:29:24.629Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 61 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "c4c387e5-3791-42e4-b8e7-ff8f0d3ebb17" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.35:48162 10.134.0.35:443 10.133.0.11:53500 - v1-completions-path 2026-11-02T22:29:40.318539Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:29:50.402378Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:29:52.016076Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T22:29:57.997Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 61 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "9b539127-5468-4cb6-aded-0aa239dba729" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.35:48162 10.134.0.35:443 10.133.0.11:42716 - v1-completions-path [2026-11-02T22:29:58.150Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 55 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "a1f71c15-a228-4097-98b2-902205380ab3" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.34:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.35:48162 10.134.0.35:443 10.133.0.11:42730 - v1-completions-path 2026-11-02T22:30:11.011336Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:30:12.612575Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:30:29.190627Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:30:31.316949Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T22:30:46.659Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 282 68 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "fe323cc4-d11c-401f-85b2-d53c0e433267" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.134.0.32:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.35:53308 10.134.0.35:443 10.133.0.11:49878 - v1-completions-path 2026-11-02T22:30:47.886726Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:30:49.616188Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:31:05.586339Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:31:16.297736Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:31:19.121628Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=4 2026-11-02T22:31:19.186587Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T22:31:33.284Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 0 38 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "a4441dab-910a-4986-a17d-2a5404907cf4" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.35:443 10.133.0.11:47868 - v1-completions-path 2026-11-02T22:31:41.316787Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:31:56.518576Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:32:06.734230Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:32:22.154648Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:32:24.417009Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:34:00.996084Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:34:03.357405Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:35:09.203914Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T22:35:09.203939Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T22:35:09.203942Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T22:35:09.203946Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T22:35:09.159Z] "POST /llm/e2e-trlp-test-simulated/v1/completions HTTP/1.1" 403 - - "-" 73 113 44 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "c4318bb3-ebd6-4a15-9644-0158c84ef827" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-trlp-test-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.35:443 10.133.0.11:46144 - v1-completions-path 2026-11-02T22:35:27.837114Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:35:42.552245Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:35:44.272602Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:35:50.093228Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T22:35:50.093257Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T22:35:50.093261Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T22:35:50.093267Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T22:35:50.059Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 33 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "9e33d9c2-94b8-4e19-9172-bbeb27723520" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.35:443 10.133.0.11:32774 - v1-completions-path 2026-11-02T22:36:01.521441Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:37:23.940093Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:37:23.940128Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:37:23.940140Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T22:37:23.940145Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-02T22:37:23.872Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 67 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "fb47e06e-125f-4d66-ab41-49395051e84f" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.35:49130 10.134.0.35:443 10.133.0.11:53074 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T22:37:24.049827Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:37:24.049859Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:37:24.049871Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T22:37:24.049876Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T22:37:23.999Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 50 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "4f352346-0de5-4412-b2e5-a30a7d9d0eba" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.133.0.31:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.35:49130 10.134.0.35:443 10.133.0.11:53080 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T22:37:27.353353Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:37:45.088591Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:38:00.143192Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:38:11.580541Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:38:14.954041Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:38:26.123532Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:38:28.441334Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 [2026-11-02T22:38:34.672Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 57 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "704d72ed-ee4b-4f57-8f64-5aea3a71bf0c" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.35:443 10.133.0.11:38900 - v1-completions-path 2026-11-02T22:38:46.352363Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:41:34.231968Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:41:37.401794Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-02T22:41:37.327Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 74 - "44.212.242.249,10.133.0.11" "python-requests/2.34.2" "1bd350d4-4151-42c2-8f00-a632b2421ce6" "maas.apps.1d4172f35d0cbb343988.prod.konflux-ocp-ci.dev" "10.134.0.30:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.35:43402 10.134.0.35:443 10.133.0.11:57580 - v1-catch-all-path 2026-11-02T22:41:39.328940Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:41:54.326578Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:41:55.889287Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:42:10.360749Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9 2026-11-02T22:42:21.048899Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=9