2026-11-03T23:38:03.023006Z info FLAG: --concurrency="0" 2026-11-03T23:38:03.023023Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-03T23:38:03.023028Z info FLAG: --help="false" 2026-11-03T23:38:03.023030Z info FLAG: --log_as_json="false" 2026-11-03T23:38:03.023032Z info FLAG: --log_caller="" 2026-11-03T23:38:03.023034Z info FLAG: --log_output_level="default:info" 2026-11-03T23:38:03.023036Z info FLAG: --log_stacktrace_level="default:none" 2026-11-03T23:38:03.023040Z info FLAG: --log_target="[stdout]" 2026-11-03T23:38:03.023043Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-03T23:38:03.023045Z info FLAG: --outlierLogPath="" 2026-11-03T23:38:03.023047Z info FLAG: --profiling="true" 2026-11-03T23:38:03.023049Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-03T23:38:03.023051Z info FLAG: --proxyLogLevel="warning" 2026-11-03T23:38:03.023053Z info FLAG: --serviceCluster="istio-proxy" 2026-11-03T23:38:03.023055Z info FLAG: --stsPort="0" 2026-11-03T23:38:03.023057Z info FLAG: --templateFile="" 2026-11-03T23:38:03.023059Z info FLAG: --tokenManagerPlugin="" 2026-11-03T23:38:03.023061Z info FLAG: --vklog="0" 2026-11-03T23:38:03.023064Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-03T23:38:03.023197Z info Proxy role ips=[10.132.0.54] type=router id=maas-default-gateway-openshift-default-778b7b5c47-m6wnb.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-03T23:38:03.023248Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-03T23:38:03.024364Z info cpu limit detected as 2, setting concurrency 2026-11-03T23:38:03.024579Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-03T23:38:03.024590Z info JWT policy is third-party-jwt 2026-11-03T23:38:03.024593Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-03T23:38:03.026246Z info Starting default Istio SDS Server 2026-11-03T23:38:03.026282Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-03T23:38:03.026300Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-03T23:38:03.026689Z info Opening status port 15020 2026-11-03T23:38:03.027811Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-03T23:38:03.028206Z info sds Starting SDS grpc server 2026-11-03T23:38:03.028243Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-03T23:38:03.028958Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-03T23:38:03.029691Z info Starting proxy agent 2026-11-03T23:38:03.029722Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-03T23:38:03.093497Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-03T23:38:03.095853Z info cache generated new workload certificate resourceName=default latency=67.499756ms ttl=23h59m59.904150586s 2026-11-03T23:38:03.095887Z info cache Root cert has changed, start rotating root cert 2026-11-03T23:38:03.095915Z info cache returned workload trust anchor from cache ttl=23h59m59.904085589s 2026-11-03T23:38:03.189552Z info ads ADS: new connection for node:1 2026-11-03T23:38:03.189624Z info cache returned workload certificate from cache ttl=23h59m59.810378446s 2026-11-03T23:38:03.189891Z info ads ADS: new connection for node:2 2026-11-03T23:38:03.189967Z info cache returned workload trust anchor from cache ttl=23h59m59.810034329s 2026-11-03T23:38:03.271698Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T23:38:11.260293Z info Readiness succeeded in 8.239775203s 2026-11-03T23:38:11.260599Z info Envoy proxy is ready [2026-11-03T23:38:41.030Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 339 82 - "34.228.250.194,10.132.0.9" "curl/7.76.1" "388f6498-9559-4638-915d-df00c3e9ba32" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.28:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:55336 10.132.0.54:443 10.132.0.9:34222 - v1-model-routing [2026-11-03T23:38:41.590Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 52 - "34.228.250.194,10.132.0.9" "curl/7.76.1" "ff7d646a-a014-46b5-b638-e172c0263c73" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.54:443 10.132.0.9:45800 - v1-model-routing [2026-11-03T23:38:41.723Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 58 - "34.228.250.194,10.132.0.9" "curl/7.76.1" "f80088cb-d49a-4139-914c-201ccc386536" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.54:443 10.132.0.9:45806 - v1-model-routing 2026-11-03T23:41:57.960640Z info ads ADS: new connection for node:3 2026-11-03T23:41:57.985394Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-03T23:41:57.987855Z info cache read certificate from file resource=file-root:system 2026-11-03T23:41:58.436983Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T23:42:03.446353Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T23:42:13.184884Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T23:42:19.409879Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:42:19.409927Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:42:19.409942Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T23:42:19.409948Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:42:19.410157Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 2026-11-03T23:42:20.105640Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T23:42:20.105676Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T23:42:20.105686Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T23:42:20.105690Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-03T23:42:20.105903Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 2026-11-03T23:42:20.228404Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T23:42:16.827Z] "POST /anything/e2e-rt-eq-0cb8b9-client HTTP/1.1" 200 - via_upstream - "-" 124 1575 2582 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "339c540a-e637-4836-b8ab-ca3e55238efd" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.54:50652 10.132.0.54:443 10.132.0.9:57450 - llm.external-model-e2e-rt-eq-0cb8b9-2304eb31a1d7.0 [2026-11-03T23:42:19.960Z] "POST /anything/e2e-rt-eq-0cb8b9-client HTTP/1.1" 200 - via_upstream - "-" 124 1576 145 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "1d665c39-8778-4aec-a58f-09dd96f057b7" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.54:50656 10.132.0.54:443 10.132.0.9:57468 - llm.external-model-e2e-rt-eq-0cb8b9-2304eb31a1d7.2 2026-11-03T23:42:25.993009Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:42:25.993052Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:42:25.993067Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:42:25.993074Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T23:42:25.993327Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T23:42:25.849Z] "POST /anything/e2e-rt-eq-0cb8b9-client HTTP/1.1" 200 - via_upstream - "-" 124 1577 143 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "3a10a397-8520-4451-97b9-dd0eb628c358" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.54:52212 10.132.0.54:443 10.132.0.9:56408 - llm.external-model-e2e-rt-eq-0cb8b9-2304eb31a1d7.0 2026-11-03T23:42:31.725150Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T23:42:37.663290Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:42:37.663405Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:42:37.663449Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T23:42:37.663477Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:42:37.663782Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T23:42:37.456Z] "POST /llm/e2e-rt-sw-59f997/anything/e2e-rt-sw-59f997-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 206 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "0815d63e-7157-4031-b6ad-9c505545dba8" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-59f997.llm.svc.cluster.local - 10.132.0.54:443 10.132.0.9:53716 - llm.external-model-e2e-rt-sw-59f997-fb96a5702750.2 2026-11-03T23:42:42.973271Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-03T23:42:42.973502Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:42:42.973524Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:42:42.973535Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:42:42.973541Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T23:42:42.973685Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-03T23:42:42.910Z] "POST /llm/e2e-rt-sw-59f997/anything/e2e-rt-sw-59f997-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 63 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "f4729f70-9d3f-472b-8056-8d6c34dce225" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-59f997.llm.svc.cluster.local - 10.132.0.54:443 10.132.0.9:33668 - llm.external-model-e2e-rt-sw-59f997-fb96a5702750.2 2026-11-03T23:42:44.913247Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T23:43:11.214871Z info ads ADS: "" 3 terminated 2026-11-03T23:43:11.277031Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T23:43:30.239470Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T23:44:59.649818Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T23:45:25.296835Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T23:46:52.904622Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:46:52.904657Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:46:52.904672Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T23:46:52.904677Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T23:46:52.829Z] "DELETE /v1/api-keys/5e393eca-1aad-4794-b7d5-1ee7b333cd64 HTTP/1.1" 200 - via_upstream - "-" 0 321 75 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "2502bef7-9a6c-4e2f-b841-f03d071f9664" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:56328 10.132.0.54:443 10.132.0.9:41888 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T23:46:55.855951Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:46:55.855986Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:46:55.855999Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T23:46:55.856004Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:46:55.973923Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:46:55.973959Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:46:55.973972Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-03T23:46:55.973977Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-03T23:46:55.796Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 59 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "2964597e-6f35-47ea-b498-2dbfa17d7207" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:56328 10.132.0.54:443 10.132.0.9:41902 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T23:46:55.915Z] "DELETE /v1/api-keys/f05d0a87-aa40-4ad2-9cd4-ecc1fa3c9333 HTTP/1.1" 200 - via_upstream - "-" 0 321 58 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "d5368cd8-24f7-4737-90de-18d4b5d24b32" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:56328 10.132.0.54:443 10.132.0.9:41908 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T23:46:56.680008Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-03T23:47:14.757Z] "POST /maas-api/internal/v1/api-keys/cleanup HTTP/1.1" 404 NR route_not_found - "-" 2 0 8 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "7702ca41-bbf8-4efd-b4f6-db04020ea9fc" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "-" - - 10.132.0.54:443 10.132.0.9:35344 - - 2026-11-03T23:47:27.265831Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:47:27.265874Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:47:27.265890Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-03T23:47:27.265895Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-03T23:47:27.209Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 605 56 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "befd69a3-c71d-4abb-8ec3-ac815c4b7991" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:56328 10.132.0.54:443 10.132.0.9:51806 - odh-ai-gateway-infra.maas-api-route.3 2026-11-03T23:48:10.349343Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-03T23:50:42.647140Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T23:50:42.647174Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T23:50:42.647187Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-03T23:50:42.647192Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-03T23:50:42.583Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 72 314 64 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "5eb5003d-d81a-47e7-a724-041506151049" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:37586 10.132.0.54:443 10.132.0.9:56002 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-03T23:52:39.587Z] "POST /maas-api/v1/tokens HTTP/1.1" 401 - - "-" 20 88 23 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "273aeb2b-4920-4204-9797-92d018b189ba" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "-" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local - 10.132.0.54:443 10.132.0.9:38516 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T00:01:54.229719Z info ads ADS: new connection for node:4 2026-11-04T00:01:54.233240Z info cache read certificate from file resource=file-root:system 2026-11-04T00:06:40.874928Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-04T00:10:59.681308Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:11:29.493258Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:12:02.242553Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:12:24.332351Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:12:51.380426Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:13:11.643999Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:13:49.906311Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:14:15.870069Z info ads ADS: new connection for node:5 2026-11-04T00:14:15.876038Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-04T00:14:15.876086Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-04T00:14:44.804381Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T00:14:53.878Z] "POST /llm/e2e-dft-9799df/anything/e2e-dft-9799df-920567 HTTP/1.1" 401 - - "-" 109 88 49 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "38d45f37-f103-458b-b4c6-de8af2803c32" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-9799df.llm.svc.cluster.local - 10.132.0.54:443 10.132.0.9:50292 - llm.external-model-e2e-dft-9799df-f642faaf8209.2 2026-11-04T00:15:05.685646Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:15:24.515244Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:17:50.484835Z info ads ADS: "" 5 terminated 2026-11-04T00:19:26.596495Z info ads ADS: "" 4 terminated 2026-11-04T00:24:45.332783Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:26:47.139061Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-11-04T00:27:57.113573Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:28:31.221038Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:29:53.106120Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:36:16.626437Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-11-04T00:39:10.243724Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T00:39:10.243781Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T00:39:10.243795Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T00:39:10.243801Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-04T00:39:10.175Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 30 314 68 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "11955142-f7b9-46e9-b4d0-be0f053db6aa" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:32928 10.132.0.54:443 10.132.0.9:43242 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T00:39:12.447157Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:39:35.197003Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:39:47.586823Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T00:39:47.586857Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T00:39:47.586871Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T00:39:47.586877Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-04T00:39:47.522Z] "POST /v1/api-keys HTTP/1.1" 400 - via_upstream - "-" 80 94 64 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "144e49ee-ad00-485a-9586-dcf05a1d1c7e" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:32928 10.132.0.54:443 10.132.0.9:36476 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T00:40:08.889041Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T00:40:13.071Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 140 405 75 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "34b41b7b-fb09-4dfc-b359-646e80ffcf64" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.133.0.43:8000" outbound|8000||echo-simulator-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:45986 10.132.0.54:443 10.132.0.9:48360 - v1-chat-completions-path 2026-11-04T00:40:24.675935Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:40:26.481666Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:40:33.004573Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:40:35.853873Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-04T00:40:35.884260Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=24 [2026-11-04T00:40:35.766Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 61 4185 117 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "0c8341ce-b1af-4fb5-9bc4-f8461ae0dfd8" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.26:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:52580 10.132.0.54:443 10.132.0.9:38898 - v1-catch-all-path 2026-11-04T00:40:48.175986Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:40:49.881411Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:41:04.402153Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:41:15.417989Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:41:17.693264Z info ads ADS: new connection for node:6 2026-11-04T00:41:17.693508Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-04T00:41:17.693577Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-04T00:41:19.815651Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:41:59.469585Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T00:42:04.146Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 131 327 85 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "3842dd0f-d31b-452e-b7f5-6b1c2dae8725" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.133.0.202:8000" outbound|54321||e2e-prio-pool-4fbc73a8-inference-pool-ip-89c5d4d5.llm.svc.cluster.local 10.132.0.54:53760 10.132.0.54:443 10.132.0.9:36188 - v1-model-routing 2026-11-04T00:42:28.650246Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:42:38.760498Z info ads ADS: "" 6 terminated 2026-11-04T00:42:40.734272Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:42:43.162252Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:42:54.344358Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:45:23.162410Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:45:39.531438Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:45:53.166091Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T00:45:53.166133Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T00:45:53.166151Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T00:45:53.166157Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T00:45:53.269527Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T00:45:53.269566Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T00:45:53.269581Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T00:45:53.269586Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-04T00:45:53.105Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 80 322 60 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "175ccae3-0eb0-4b9f-bfc4-7763bdf66d4d" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:40076 10.132.0.54:443 10.132.0.9:56126 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-04T00:45:53.222Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 628 47 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "2a962dc4-b916-4bcc-9aae-50af2e2d7a45" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:40076 10.132.0.54:443 10.132.0.9:56142 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T00:45:54.648348Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:46:06.387867Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:46:10.119820Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:46:29.182356Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:46:31.589679Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:46:46.861389Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T00:47:27.012Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 285 72 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "c44b1d7f-9577-4bd0-b9bf-8d8cf524f631" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.28:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:48966 10.132.0.54:443 10.132.0.9:34140 - v1-completions-path 2026-11-04T00:48:02.097099Z info ads ADS: new connection for node:7 2026-11-04T00:48:02.101684Z info cache read certificate from file resource=file-root:system 2026-11-04T00:48:02.766517Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:48:04.782699Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:48:37.057975Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T00:48:37.058042Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T00:48:37.058086Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T00:48:37.058102Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T00:48:37.058337Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-04T00:48:36.933Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1536 124 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "34fd32ef-05ee-469b-8359-0b1225ee15f9" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.54:33774 10.132.0.54:443 10.132.0.9:36982 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-04T00:48:39.239837Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T00:48:39.239866Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T00:48:39.239876Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T00:48:39.239881Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T00:48:39.240031Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-04T00:48:39.123Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1536 116 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "675121a8-23fe-49b0-ae78-08b1343fc726" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.54:33784 10.132.0.54:443 10.132.0.9:37002 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-04T00:48:54.479310Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T00:48:54.479343Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T00:48:54.479354Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T00:48:54.479359Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-04T00:48:54.479519Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-04T00:48:54.383Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1530 96 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "85265160-c8b7-4889-b737-ac89e84556f2" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.132.0.54:33774 10.132.0.54:443 10.132.0.9:33900 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-04T00:49:40.489456Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:49:40.611885Z info ads ADS: "" 7 terminated with stream closed 2026-11-04T00:49:41.079225Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:49:43.818377Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:49:45.627119Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:50:01.231900Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:50:03.080505Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:50:18.120893Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:50:19.862921Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:50:22.630695Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:50:31.091022Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T00:50:38.042Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 71 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "f81762c3-7305-4217-b9ce-c4d70595e901" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.27:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:39172 10.132.0.54:443 10.132.0.9:57822 - v1-completions-path [2026-11-04T00:50:38.389Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 54 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "36711fdc-b513-4a85-9dea-7898893f4331" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.27:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:39172 10.132.0.54:443 10.132.0.9:57832 - v1-completions-path [2026-11-04T00:50:39.090Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 59 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "61eae440-6657-46a7-800c-5dc447c553e1" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.27:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:39172 10.132.0.54:443 10.132.0.9:57846 - v1-completions-path [2026-11-04T00:50:39.944Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 60 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "1928d5ff-0e27-4a28-a964-edc93312fdbc" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.27:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:39182 10.132.0.54:443 10.132.0.9:57856 - v1-completions-path 2026-11-04T00:50:55.704763Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:51:05.978450Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:51:07.798702Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T00:51:14.507Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 55 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "421e6cc4-40c1-4a20-bbf0-7c5eda4f3507" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.27:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:39182 10.132.0.54:443 10.132.0.9:55888 - v1-completions-path 2026-11-04T00:51:26.788588Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:51:28.497807Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:51:34.152366Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:51:45.111213Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:51:58.986814Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:52:00.729721Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:52:06.337308Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:52:24.329532Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:52:35.091279Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:52:49.429436Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:52:51.187984Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-04T00:52:52.439Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 280 63 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "03a0bcfe-03b4-43b9-92c2-931bc21ab708" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.28:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.132.0.54:48966 10.132.0.54:443 10.132.0.9:59208 - v1-completions-path 2026-11-04T00:52:57.054434Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:53:10.036450Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T00:53:10.036488Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T00:53:10.036506Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-04T00:53:10.036512Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-04T00:53:09.978Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 65 308 57 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "5bde7886-3f12-4075-bd50-b2741139a049" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.24:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:60124 10.132.0.54:443 10.132.0.9:50378 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T00:53:22.004593Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:53:32.266000Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:54:19.275400Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:54:22.181626Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:57:08.799988Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:57:11.621125Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:58:31.899495Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:58:55.956691Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:58:57.717914Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:59:03.610650Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:59:22.441125Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:59:26.446062Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:59:42.035515Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T00:59:52.412007Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:00:04.856845Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:00:10.715900Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:00:10.715937Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:00:10.715954Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-04T01:00:10.715961Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-04T01:00:10.655Z] "DELETE /v1/api-keys/584bfd2b-cda6-41a8-8f14-0dbc17c25f49 HTTP/1.1" 200 - via_upstream - "-" 0 365 60 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "cbb70075-795f-487c-8f89-4f163b195c19" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.133.0.245:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:48866 10.132.0.54:443 10.132.0.9:59288 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:00:22.432031Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:00:34.501703Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:00:38.510527Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:00:48.399342Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:00:48.399376Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:00:48.399389Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-04T01:00:48.399394Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-04T01:00:48.300Z] "DELETE /v1/api-keys/4ee9992e-9680-4055-bf28-f7d025b71326 HTTP/1.1" 200 - via_upstream - "-" 0 365 98 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "dd92042d-0e1f-4bf8-8fa6-470b8941dc3a" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "10.134.0.243:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.132.0.54:35676 10.132.0.54:443 10.132.0.9:35256 - odh-ai-gateway-infra.maas-api-route.3 2026-11-04T01:01:00.060923Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:01:10.205245Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:01:12.478602Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:01:15.592549Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:01:21.804648Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:01:38.265298Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:01:48.391518Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:01:50.128024Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:02:10.498816Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:02:13.471806Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:02:16.762656Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:02:32.777983Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:02:52.732838Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:03:02.915941Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:03:13.175999Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:03:23.469558Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:03:33.638016Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:03:43.959794Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:03:54.096023Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:04:04.399803Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:04:14.579651Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:04:24.782654Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-04T01:04:38.138867Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-04T01:04:38.138897Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-04T01:04:38.138901Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-04T01:04:38.138905Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-04T01:04:38.112Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 26 - "34.228.250.194,10.132.0.9" "python-requests/2.34.2" "9052469c-f1ad-40ae-bf28-4c9e3f518855" "maas.apps.b4341b0d6c25eae239a7.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.132.0.54:443 10.132.0.9:33668 - v1-completions-path 2026-11-04T01:05:24.962885Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=4