2026-11-02T17:40:22.635219Z info FLAG: --concurrency="0" 2026-11-02T17:40:22.635236Z info FLAG: --domain="openshift-ingress.svc.cluster.local" 2026-11-02T17:40:22.635241Z info FLAG: --help="false" 2026-11-02T17:40:22.635243Z info FLAG: --log_as_json="false" 2026-11-02T17:40:22.635245Z info FLAG: --log_caller="" 2026-11-02T17:40:22.635247Z info FLAG: --log_output_level="default:info" 2026-11-02T17:40:22.635249Z info FLAG: --log_stacktrace_level="default:none" 2026-11-02T17:40:22.635254Z info FLAG: --log_target="[stdout]" 2026-11-02T17:40:22.635256Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2026-11-02T17:40:22.635258Z info FLAG: --outlierLogPath="" 2026-11-02T17:40:22.635260Z info FLAG: --profiling="true" 2026-11-02T17:40:22.635262Z info FLAG: --proxyComponentLogLevel="misc:error" 2026-11-02T17:40:22.635264Z info FLAG: --proxyLogLevel="warning" 2026-11-02T17:40:22.635266Z info FLAG: --serviceCluster="istio-proxy" 2026-11-02T17:40:22.635268Z info FLAG: --stsPort="0" 2026-11-02T17:40:22.635270Z info FLAG: --templateFile="" 2026-11-02T17:40:22.635272Z info FLAG: --tokenManagerPlugin="" 2026-11-02T17:40:22.635274Z info FLAG: --vklog="0" 2026-11-02T17:40:22.635277Z info Version 1.28.10-cebc47d567fd1333707199224c18129ecfe07c05-Clean 2026-11-02T17:40:22.635426Z info Proxy role ips=[10.134.0.41] type=router id=maas-default-gateway-openshift-default-65c448b547-jv8f4.openshift-ingress domain=openshift-ingress.svc.cluster.local 2026-11-02T17:40:22.635476Z info Apply proxy config from env {"discoveryAddress":"istiod-openshift-gateway.openshift-ingress.svc:15012","proxyHeaders":{"server":{"disabled":true},"envoyDebugHeaders":{"disabled":true},"metadataExchangeHeaders":{"mode":"IN_MESH"}}} 2026-11-02T17:40:22.636551Z info cpu limit detected as 2, setting concurrency 2026-11-02T17:40:22.636819Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 2 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-openshift-gateway.openshift-ingress.svc:15012 drainDuration: 45s proxyAdminPort: 15000 proxyHeaders: envoyDebugHeaders: disabled: true metadataExchangeHeaders: mode: IN_MESH server: disabled: true serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s 2026-11-02T17:40:22.636831Z info JWT policy is third-party-jwt 2026-11-02T17:40:22.636834Z info using credential fetcher of JWT type in cluster.local trust domain 2026-11-02T17:40:22.638084Z info Starting default Istio SDS Server 2026-11-02T17:40:22.638106Z info CA Endpoint istiod-openshift-gateway.openshift-ingress.svc:15012, provider Citadel 2026-11-02T17:40:22.638122Z info Using CA istiod-openshift-gateway.openshift-ingress.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2026-11-02T17:40:22.638514Z info Opening status port 15020 2026-11-02T17:40:22.639606Z info xdsproxy Initializing with upstream address "istiod-openshift-gateway.openshift-ingress.svc:15012" and cluster "Kubernetes" 2026-11-02T17:40:22.640111Z info sds Starting SDS grpc server 2026-11-02T17:40:22.640126Z info sds Starting SDS server for workload certificates, will listen on "var/run/secrets/workload-spiffe-uds/socket" 2026-11-02T17:40:22.640856Z info Pilot SAN: [istiod-openshift-gateway.openshift-ingress.svc] 2026-11-02T17:40:22.641642Z info Starting proxy agent 2026-11-02T17:40:22.641672Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --skip-deprecated-logs --concurrency 2] 2026-11-02T17:40:22.704896Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=1 2026-11-02T17:40:22.714042Z info cache generated new workload certificate resourceName=default latency=73.820894ms ttl=23h59m59.28596125s 2026-11-02T17:40:22.714076Z info cache Root cert has changed, start rotating root cert 2026-11-02T17:40:22.714113Z info cache returned workload trust anchor from cache ttl=23h59m59.285888021s 2026-11-02T17:40:22.813685Z info ads ADS: new connection for node:1 2026-11-02T17:40:22.813741Z info cache returned workload certificate from cache ttl=23h59m59.186260509s 2026-11-02T17:40:22.813983Z info ads ADS: new connection for node:2 2026-11-02T17:40:22.814059Z info cache returned workload trust anchor from cache ttl=23h59m59.185942585s 2026-11-02T17:40:22.894592Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:40:30.745667Z info Readiness succeeded in 8.11344042s 2026-11-02T17:40:30.746093Z info Envoy proxy is ready 2026-11-02T17:40:33.696520Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:40:33.696564Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:40:33.696580Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:40:33.696585Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T17:40:33.600Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 567 95 - "44.212.242.249,10.132.0.15" "curl/7.76.1" "075ad081-a3cf-48bd-8621-eddc052dbbab" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.41:59572 10.134.0.41:443 10.132.0.15:38604 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T17:40:33.943Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 337 81 - "44.212.242.249,10.132.0.15" "curl/7.76.1" "92daad0f-1864-476c-9706-4e6466aceabc" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:34986 10.134.0.41:443 10.132.0.15:38608 - v1-model-routing [2026-11-02T17:40:34.094Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 330 59 - "44.212.242.249,10.132.0.15" "curl/7.76.1" "3a08253f-4b10-448d-84b1-671db7587779" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:34998 10.134.0.41:443 10.132.0.15:38614 - v1-model-routing [2026-11-02T17:40:34.212Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 122 509 53 - "44.212.242.249,10.132.0.15" "curl/7.76.1" "aff0de47-c438-4532-a811-11ff3215c6ac" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:34998 10.134.0.41:443 10.132.0.15:38628 - v1-model-routing 2026-11-02T17:40:35.264336Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:40:35.264371Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:40:35.264385Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:40:35.264390Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 [2026-11-02T17:40:34.790Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 53 - "44.212.242.249,10.132.0.15" "curl/7.76.1" "6b3544ab-3fa5-47ac-b15f-7fd784623d89" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.41:443 10.132.0.15:38642 - v1-model-routing [2026-11-02T17:40:34.906Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 54 - "44.212.242.249,10.132.0.15" "curl/7.76.1" "dd95fe74-91be-4c1f-b036-537075994e28" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.41:443 10.132.0.15:38662 - v1-model-routing [2026-11-02T17:40:35.022Z] "POST /v1/chat/completions HTTP/1.1" 429 - - "-" 122 18 50 - "44.212.242.249,10.132.0.15" "curl/7.76.1" "32ff657a-2c69-4744-a511-3ae8d52ba40c" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.41:443 10.132.0.15:38672 - v1-model-routing [2026-11-02T17:40:35.208Z] "DELETE /v1/api-keys/1e48c29b-4c0d-408c-8d93-94a7a4262ebf HTTP/1.1" 200 - via_upstream - "-" 0 365 55 - "44.212.242.249,10.132.0.15" "curl/7.76.1" "20b7c79f-27eb-446e-a40e-cedb867e4916" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.41:59572 10.134.0.41:443 10.132.0.15:38680 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T17:40:35.788521Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:40:35.788557Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:40:35.788572Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:40:35.788577Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 [2026-11-02T17:40:35.731Z] "GET /v1/subscriptions HTTP/1.1" 200 - via_upstream - "-" 0 488 57 - "44.212.242.249,10.132.0.15" "curl/7.76.1" "d8e8e7fd-18e9-486f-b053-a52dd13593f6" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.41:59586 10.134.0.41:443 10.132.0.15:38682 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T17:43:21.405391Z info ads ADS: new connection for node:3 2026-11-02T17:43:21.409450Z info cache adding watcher for file certificate /etc/pki/tls/certs/ca-bundle.crt 2026-11-02T17:43:21.409498Z info cache read certificate from file resource=file-root:system 2026-11-02T17:43:21.886229Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:43:27.162761Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:43:37.971641Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:43:47.900327Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:43:52.481163Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:43:52.481197Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:43:52.481208Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:43:52.481212Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T17:43:52.481401Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 2026-11-02T17:43:52.650331Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:43:52.650362Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:43:52.650379Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:43:52.650384Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T17:43:52.650545Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T17:43:52.334Z] "POST /anything/e2e-rt-sw-9931ad-client HTTP/1.1" 200 - via_upstream - "-" 124 1763 147 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "d5a6bd7f-be7d-4595-b2d1-f82269c2074c" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:48496 10.134.0.41:443 10.132.0.15:55420 - llm.external-model-e2e-rt-sw-9931ad-23637e980040.0 [2026-11-02T17:43:52.534Z] "POST /anything/e2e-rt-sw-9931ad-client HTTP/1.1" 200 - via_upstream - "-" 124 1768 116 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "1c1acd13-c937-488c-a8d9-84f97dac17f7" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:48496 10.134.0.41:443 10.132.0.15:55426 - llm.external-model-e2e-rt-sw-9931ad-23637e980040.0 2026-11-02T17:43:59.320963Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:43:59.321001Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:43:59.321017Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T17:43:59.321023Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:43:59.321293Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T17:43:58.916Z] "POST /anything/e2e-rt-sw-9931ad-client HTTP/1.1" 200 - via_upstream - "-" 124 1768 404 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "676c0bb3-3ae1-4923-b13b-cdf7d231568a" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:55630 10.134.0.41:443 10.132.0.15:50816 - llm.external-model-e2e-rt-sw-9931ad-23637e980040.0 2026-11-02T17:44:04.788689Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:44:04.788720Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:44:04.788730Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:44:04.788735Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:44:04.788891Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:44:04.724Z] "POST /llm/e2e-rt-sw-9931ad/anything/e2e-rt-sw-9931ad-client HTTP/1.1" 503 UH no_healthy_upstream - "-" 124 19 64 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "f1799eb5-1ef5-4cac-b285-63bb4da39810" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rt-sw-9931ad.llm.svc.cluster.local - 10.134.0.41:443 10.132.0.15:43810 - llm.external-model-e2e-rt-sw-9931ad-23637e980040.2 2026-11-02T17:44:08.600738Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:44:18.261732Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:44:18.261776Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:44:18.261792Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:44:18.261799Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:44:18.262026Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:44:18.121Z] "POST /anything/e2e-rt-sw-9931ad-client HTTP/1.1" 200 - via_upstream - "-" 124 1767 140 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "18b90928-6b61-44ae-86a4-599007a8b8be" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:41050 10.134.0.41:443 10.132.0.15:43726 - llm.external-model-e2e-rt-sw-9931ad-23637e980040.0 2026-11-02T17:44:33.905298Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:44:44.115274Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:45:00.153609Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=24 2026-11-02T17:45:00.154129Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:45:00.154157Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:45:00.154173Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T17:45:00.154180Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:45:00.154469Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 2026-11-02T17:45:00.395103Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:45:00.395149Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:45:00.395167Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:45:00.395174Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:45:00.395464Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:44:59.995Z] "POST /anything/e2e-rt-eq-b76f7a-client HTTP/1.1" 200 - via_upstream - "-" 124 1767 158 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "4b265102-83cc-4522-822a-c181b1ba7f15" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:48834 10.134.0.41:443 10.132.0.15:43670 - llm.external-model-e2e-rt-eq-b76f7a-c2af0e331308.2 [2026-11-02T17:45:00.211Z] "POST /anything/e2e-rt-eq-b76f7a-client HTTP/1.1" 200 - via_upstream - "-" 124 1762 183 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "9b11232e-9223-473b-96a4-34021307b5c1" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:48848 10.134.0.41:443 10.132.0.15:43686 - llm.external-model-e2e-rt-eq-b76f7a-c2af0e331308.2 2026-11-02T17:45:00.629052Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Tracing span buffer full (100), dropping oldest spans to prevent memory growth thread=25 2026-11-02T17:45:00.629697Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:45:00.629783Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:45:00.629832Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:45:00.629863Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:45:00.630189Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:45:00.470Z] "POST /anything/e2e-rt-eq-b76f7a-client HTTP/1.1" 200 - via_upstream - "-" 124 1766 159 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "773f92e5-88d6-4245-bc88-1eecac127ced" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:48860 10.134.0.41:443 10.132.0.15:43698 - llm.external-model-e2e-rt-eq-b76f7a-c2af0e331308.0 2026-11-02T17:45:03.594374Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:45:03.594515Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:45:03.594569Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:45:03.594600Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:45:03.594977Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:45:00.960Z] "POST /anything/e2e-rt-eq-b76f7a-client HTTP/1.1" 200 - via_upstream - "-" 124 1770 2634 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "024db614-6d94-43fc-900e-3392da8c4434" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:48864 10.134.0.41:443 10.132.0.15:43712 - llm.external-model-e2e-rt-eq-b76f7a-c2af0e331308.0 2026-11-02T17:45:05.664890Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:45:05.664928Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:45:05.664945Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:45:05.664951Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:45:05.665195Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T17:45:05.327Z] "POST /anything/e2e-rt-eq-b76f7a-client HTTP/1.1" 200 - via_upstream - "-" 124 1760 337 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "dfce98f3-6739-480b-8ad5-743abfb57859" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:48860 10.134.0.41:443 10.132.0.15:43324 - llm.external-model-e2e-rt-eq-b76f7a-c2af0e331308.2 2026-11-02T17:45:24.351395Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:45:30.507039Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:45:30.545227Z info ads ADS: "" 3 terminated with stream closed 2026-11-02T17:45:40.806673Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:46:11.658267Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:46:28.005798Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:47:45.751378Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:48:12.902253Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T17:48:20.694071Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:48:20.694107Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T17:48:20.694122Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T17:48:20.694127Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-02T17:48:20.626Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 314 67 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "49e9afb6-ebcf-4009-bf27-9119a56f5398" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.41:41880 10.134.0.41:443 10.132.0.15:41308 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T17:48:34.883821Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:48:34.883863Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:48:34.883884Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T17:48:34.883891Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T17:48:35.591827Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T17:48:34.797Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 71 313 86 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "162e04c8-6558-4260-bdf2-ad0760f34be7" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.41:59054 10.134.0.41:443 10.132.0.15:45058 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T17:48:55.889816Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T17:53:13.223Z] "POST /v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 136 340 87 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "561d9e31-b60f-41d6-ad02-4ccff0fd9ac1" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:34986 10.134.0.41:443 10.132.0.15:52778 - v1-chat-completions-path 2026-11-02T18:02:30.804420Z info ads ADS: new connection for node:4 2026-11-02T18:02:30.856790Z info cache read certificate from file resource=file-root:system 2026-11-02T18:09:43.408739Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:09:44.749908Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=2 2026-11-02T18:09:59.308402Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:10:36.568318Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:11:37.031136Z info ads ADS: new connection for node:5 2026-11-02T18:11:37.031308Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T18:11:37.031339Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T18:11:41.518260Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:12:40.529036Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:13:39.918480Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:15:03.723674Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:15:13.961Z] "POST /llm/e2e-dft-32beb1/anything/e2e-dft-32beb1-c70e40 HTTP/1.1" 401 - - "-" 109 88 86 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "da460161-a0f6-42f9-a77e-ba923aa4dd03" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-dft-32beb1.llm.svc.cluster.local - 10.134.0.41:443 10.132.0.15:59538 - llm.external-model-e2e-dft-32beb1-8faa743806da.2 2026-11-02T18:15:25.854012Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:16:02.539341Z info ads ADS: "" 5 terminated with stream closed 2026-11-02T18:16:19.907282Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:19:23.013758Z info ads ADS: "" 4 terminated 2026-11-02T18:22:34.724374Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:24:30.074534Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:24:51.418665Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:26:38.065676Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:29:03.459795Z info cache event for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt : REMOVE, pushing to proxy 2026-11-02T18:34:51.468179Z info xdsproxy connected to delta upstream XDS server: istiod-openshift-gateway.openshift-ingress.svc:15012 id=3 2026-11-02T18:36:10.969343Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:36:33.379665Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:05.123276Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:20.959054Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:22.575085Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:25.839086Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'completion_tokens' missing from lines 1-1 key: completion_tokens thread=25 [2026-11-02T18:37:25.761Z] "POST /v1/embeddings HTTP/1.1" 200 - via_upstream - "-" 91 3616 77 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "338b3a4c-2f74-4ffa-ad8e-a693bae3b500" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.46:8000" outbound|8000||e2e-embedding-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:40932 10.134.0.41:443 10.132.0.15:37022 - v1-catch-all-model-routing 2026-11-02T18:37:28.028101Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:43.233307Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:44.958454Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:37:59.121603Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:38:09.926092Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:38:11.696291Z info ads ADS: new connection for node:6 2026-11-02T18:38:11.696477Z info cache adding watcher for file certificate /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T18:38:11.696506Z info cache read certificate from file resource=file-root:/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt 2026-11-02T18:38:13.028783Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:38:53.568394Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:39:12.210370Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:39:12.628056Z info ads ADS: "" 6 terminated 2026-11-02T18:39:14.015829Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:39:16.396819Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:39:20.710734Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:41:33.187475Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:41:49.758827Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:42:06.065006Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:42:17.739081Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:42:18.458146Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:42:18.458195Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:42:18.458215Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T18:42:18.458222Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-02T18:42:18.359Z] "GET /v1/models HTTP/1.1" 200 - via_upstream - "-" 0 27 98 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "8a6e4e4d-250a-4e8a-9615-7c018711af9c" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.41:42096 10.134.0.41:443 10.132.0.15:34996 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T18:42:20.687533Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:42:29.856094Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:42:32.070022Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:42:35.057Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 77 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "0eb9ce7b-f829-40b4-8b8d-f5647af1f66f" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:48326 10.134.0.41:443 10.132.0.15:48052 - v1-completions-path 2026-11-02T18:42:47.043318Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:43:31.246734Z info ads ADS: new connection for node:7 2026-11-02T18:43:31.253900Z info cache read certificate from file resource=file-root:system 2026-11-02T18:43:31.714141Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:43:33.255652Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:43:42.571575Z warning envoy ext_proc external/envoy/source/extensions/filters/http/ext_proc/ext_proc.cc:1688 [Tags: "ConnectionId":"1009","StreamId":"15700468023399655000"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}} thread=24 [2026-11-02T18:43:42.508Z] "POST /models-as-a-service/e2e-rotation-external-model/anything/v1/chat/completions HTTP/1.1" 500 - ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_delayed_connect_error:_Connection_refused{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}} - "-" 92 0 62 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "8866576f-75f8-415c-aa19-4c77519e2168" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|443||provider-selection-required-e2e-rotation-external-model.models-as-a-service.svc.cluster.local - 10.134.0.41:443 10.132.0.15:40252 - models-as-a-service.external-model-e2e-rotation-external-model.2 2026-11-02T18:43:55.948278Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:43:55.948315Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:43:55.948327Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:43:55.948333Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T18:43:55.948515Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T18:43:55.822Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1719 125 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "f174d160-723e-4993-a926-6534276eba9f" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:59122 10.134.0.41:443 10.132.0.15:44738 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:44:00.329851Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:44:00.329884Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:44:00.329896Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:44:00.329900Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T18:44:00.330066Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T18:44:00.203Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1714 126 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "843af62d-8138-4cb9-b569-dda8b3ab61e1" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:59132 10.134.0.41:443 10.132.0.15:44758 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:44:13.631065Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:44:13.631100Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:44:13.631111Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:44:13.631116Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T18:44:13.631297Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T18:44:11.129Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1718 2501 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "6a540b6d-ecc1-4caa-88e1-68cdd87064e8" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:59132 10.134.0.41:443 10.132.0.15:49668 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:44:35.367030Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:44:35.367066Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:44:35.367078Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:44:35.367083Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T18:44:35.367250Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T18:44:35.268Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1718 98 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "21199599-3394-436d-b081-cf78d0a6e3cb" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:59132 10.134.0.41:443 10.132.0.15:59004 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:44:44.067594Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:44:44.067655Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:44:44.067668Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:44:44.067672Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T18:44:44.067862Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T18:44:43.969Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1712 98 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "3cc7e767-8a20-4346-914b-7271cdc57d57" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:59122 10.134.0.41:443 10.132.0.15:38148 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:44:52.762919Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:44:52.762958Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:44:52.762971Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:44:52.762976Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 2026-11-02T18:44:52.763141Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=25 [2026-11-02T18:44:52.663Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1713 99 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "bf1d8fe8-d631-4838-ae36-54a89c621a32" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:59132 10.134.0.41:443 10.132.0.15:38166 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:45:05.801896Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:45:05.801929Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:45:05.801943Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T18:45:05.801948Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:45:05.802109Z error envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1155 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Task failed: Some("2") thread=24 [2026-11-02T18:45:05.701Z] "POST /anything/v1/chat/completions HTTP/1.1" 200 - via_upstream - "-" 92 1719 101 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "f72ee1fa-e33a-423a-b972-86ad24f3e05a" "httpbingo.org" "66.241.125.232:443" outbound|443||httpbingo.org 10.134.0.41:59122 10.134.0.41:443 10.132.0.15:39134 - models-as-a-service.external-model-e2e-rotation-external-model.0 2026-11-02T18:45:23.122181Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:45:23.269764Z info ads ADS: "" 7 terminated with stream closed 2026-11-02T18:45:23.725389Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:45:24.099388Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T18:45:24.099422Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T18:45:24.099427Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=24 2026-11-02T18:45:24.099432Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=24 [2026-11-02T18:45:24.053Z] "POST /llm/premium-simulated-simulated-premium/v1/completions HTTP/1.1" 403 - - "-" 74 77 46 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "2296e9a2-00a3-49f4-9251-2c1895940a0c" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.41:443 10.132.0.15:56626 - v1-completions-path 2026-11-02T18:45:26.266737Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:45:28.561075Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:45:31.486Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 77 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "71d1d0f2-d2eb-4742-8c2a-b33bfece00c9" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:44330 10.134.0.41:443 10.132.0.15:56650 - v1-completions-path 2026-11-02T18:45:44.176450Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:45:45.793837Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:45:48.567Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 63 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "b7e8240c-7e04-4e0d-a388-1451b7e38b40" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.41:443 10.132.0.15:52276 - v1-completions-path 2026-11-02T18:46:00.368484Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:46:02.080454Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:46:04.812961Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:46:13.149743Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:46:16.357Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 300 65 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "9a2ce13b-50d3-45bf-924e-567f3ee45c34" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:44330 10.134.0.41:443 10.132.0.15:44016 - v1-completions-path [2026-11-02T18:46:16.707Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 60 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "e51a1c57-c927-4c40-bdb6-d2550e194ecf" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:48326 10.134.0.41:443 10.132.0.15:44022 - v1-completions-path [2026-11-02T18:46:17.643Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 56 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "41f0fd09-1078-42b2-bafd-d0c41000c319" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:48326 10.134.0.41:443 10.132.0.15:44036 - v1-completions-path 2026-11-02T18:46:34.403076Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:46:44.610537Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:46:46.793762Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:46:52.756Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 291 63 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "3877a15a-48c0-45e0-8b1b-4f752a47b714" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:48326 10.134.0.41:443 10.132.0.15:36302 - v1-completions-path [2026-11-02T18:46:52.998Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 294 64 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "3a6ec374-77e1-4c76-9254-dd1479fec5f8" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:44330 10.134.0.41:443 10.132.0.15:53638 - v1-completions-path [2026-11-02T18:46:53.601Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 295 59 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "b7ba4e5c-0a14-4f5d-8ac1-e0f62d0c3940" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:48326 10.134.0.41:443 10.132.0.15:53654 - v1-completions-path 2026-11-02T18:47:05.339760Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:47:06.961576Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:47:09.941Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 74 288 86 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "aa29b866-caba-4b81-a644-6d61b9812cc7" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.49:8000" outbound|8000||premium-simulated-simulated-premium-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:36964 10.134.0.41:443 10.132.0.15:47878 - v1-completions-path 2026-11-02T18:47:12.315813Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:47:23.349801Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:47:36.943787Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:47:38.513994Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:47:43.901274Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:02.196527Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:04.827046Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:18.838348Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:20.624716Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:22.242502Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:48:22.242541Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:48:22.242558Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:48:22.242565Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T18:48:22.171Z] "DELETE /v1/api-keys/b410698b-6b2a-4b69-a80c-c97e1adeff26 HTTP/1.1" 404 - via_upstream - "-" 0 29 70 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "22e615a3-99b7-42b5-9a02-0bf552361d75" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.41:37068 10.134.0.41:443 10.132.0.15:42212 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T18:48:23.681Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 76 0 39 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "2401ef41-6acd-4b80-a159-5fb43705d390" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.41:443 10.132.0.15:56322 - v1-completions-path 2026-11-02T18:48:26.308796Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:41.336650Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:48:51.503758Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:49:06.888210Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:49:09.185374Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:51:22.240053Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:51:24.609371Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:52:49.039346Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:52:52.578143Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:52:52.578177Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 2026-11-02T18:52:52.578189Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=24 2026-11-02T18:52:52.578194Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=24 [2026-11-02T18:52:52.498Z] "DELETE /v1/api-keys/e2e-trlp-test-key HTTP/1.1" 404 - via_upstream - "-" 0 29 79 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "2d6409cb-b27b-450c-9551-f58cdf088e7b" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.27:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.41:42758 10.134.0.41:443 10.132.0.15:34716 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T18:53:04.030916Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:05.556889Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:08.764240Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:53:08.764265Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:53:08.764268Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:53:08.764273Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T18:53:08.730Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 403 - - "-" 66 123 33 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "c334800f-6196-4744-a124-4b1852b81ec2" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.41:443 10.132.0.15:42858 - v1-completions-path 2026-11-02T18:53:11.395023Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:30.212858Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:33.994629Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:53:34.239Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 293 136 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "bf89f934-bb7a-4e93-b0d6-3e22a1e52cfa" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:48326 10.134.0.41:443 10.132.0.15:40478 - v1-completions-path 2026-11-02T18:53:49.220297Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:53:59.640469Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:02.691731Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:06.494801Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:54:06.494836Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:54:06.494850Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:54:06.494855Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T18:54:06.428Z] "POST /v1/api-keys HTTP/1.1" 201 - via_upstream - "-" 67 309 66 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "12c4ccea-d25b-4dbe-887e-6610b1e16218" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.1.97:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.41:57952 10.134.0.41:443 10.132.0.15:41722 - odh-ai-gateway-infra.maas-api-route.3 [2026-11-02T18:54:07.461Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 68 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "d45bc0ee-09cc-477c-acfb-06bb2c12ac3a" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:44330 10.134.0.41:443 10.132.0.15:41726 - v1-completions-path 2026-11-02T18:54:19.548419Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:21.790444Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:25.146540Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 [2026-11-02T18:54:31.095Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 64 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "bb5cca0b-765a-41ee-97e4-471306479c98" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:48326 10.134.0.41:443 10.132.0.15:49630 - v1-completions-path [2026-11-02T18:54:32.825Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 76 292 63 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "d8a64953-0997-429b-8759-2eb828071303" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.132.0.47:8000" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:44330 10.134.0.41:443 10.132.0.15:49644 - v1-completions-path [2026-11-02T18:54:35.259Z] "POST /llm/e2e-unconfigured-facebook-opt-125m-simulated/v1/completions HTTP/1.1" 429 - - "-" 76 18 58 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "1486760a-6605-4ce3-845a-ba14576b88e4" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||e2e-unconfiab60ef4d3a239b5143b412cab04acac3-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.41:443 10.132.0.15:56492 - v1-completions-path 2026-11-02T18:54:47.112173Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:57.290390Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:54:59.308909Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:02.534983Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:07.838546Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:12.533041Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:55:12.533079Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:55:12.533094Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /model thread=25 2026-11-02T18:55:12.533099Z warning envoy wasm external/envoy/source/extensions/common/wasm/context.cc:1152 wasm log kuadrant-wasm-shim kuadrant_wasm_shim: Missing json property: /usage/total_tokens thread=25 [2026-11-02T18:55:12.472Z] "DELETE /v1/api-keys/1e3c2bcf-31c5-4d67-971d-fe781c3953bd HTTP/1.1" 200 - via_upstream - "-" 0 367 60 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "c8cf4c6a-ba1a-4ff8-bbb4-43b6eeda483b" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.1.97:8443" outbound|8443||maas-api.odh-ai-gateway-infra.svc.cluster.local 10.134.0.41:55440 10.134.0.41:443 10.132.0.15:50020 - odh-ai-gateway-infra.maas-api-route.3 2026-11-02T18:55:24.005871Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:34.148153Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:35.701924Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:43.011403Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:45.896081Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:55:49.151472Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:05.114474Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:25.015873Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:35.254627Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:45.499824Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:56:55.710785Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:05.875943Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:16.161208Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:26.330010Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:36.588612Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:46.775235Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:57.017320Z warning envoy wasm external/envoy/source/extensions/common/wasm/wasm.cc:317 Wasm remote code fetch is unstable and may cause a crash thread=8 2026-11-02T18:57:59.859436Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:57:59.859465Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:57:59.859469Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:372 thread=25 2026-11-02T18:57:59.859474Z warning envoy filter external/envoy/source/extensions/filters/http/json_to_metadata/filter.cc:385 key 'model' missing from lines 1-1 key: model thread=25 [2026-11-02T18:57:59.828Z] "POST /llm/facebook-opt-125m-simulated/v1/completions HTTP/1.1" 401 - - "-" 66 88 31 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "db73a8c4-ce63-4968-aa37-fdc4b2da2b73" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "-" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local - 10.134.0.41:443 10.132.0.15:41486 - v1-completions-path [2026-11-02T18:58:06.477Z] "POST /v1/completions HTTP/1.1" 200 - via_upstream - "-" 66 289 86 - "44.212.242.249,10.132.0.15" "python-requests/2.34.2" "d706761f-b92e-483f-82eb-b5d949c50b24" "maas.apps.0069df05874e3f91ac88.prod.konflux-ocp-ci.dev" "10.133.0.35:8000" outbound|8000||facebook-opt-125m-simulated-kserve-workload-svc.llm.svc.cluster.local 10.134.0.41:49804 10.134.0.41:443 10.132.0.15:49174 - v1-completions-path